forum.vmspython.org Forum Index forum.vmspython.org
Forum system
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Still problems with Rdb and PyRTE
Goto page 1, 2, 3, 4  Next
 
Post new topic   Reply to topic    forum.vmspython.org Forum Index -> Python for OpenVMS
View previous topic :: View next topic  
Author Message
jescab



Joined: 28 Jan 2008
Posts: 254

PostPosted: Sun May 09, 2010 6:51 pm    Post subject: Still problems with Rdb and PyRTE Reply with quote

Hi.

I still have som e troubles using PyRTE and the Rdb interface.

I have two scripts, scr1.py and scr2.py.

scr1.py displayes a simple form (two date fields and a submit button).
When submitted scr1.py displayes the result in a table.

One of the fields in the reult-table is a link to scr2.py using a few
of the table cols as URL-coded variables. scr2.py then displays
details about that particular row in the table in a saparate window.
So far so good. I can click on any number of rows in the scr1.py
table and the details are displayed in the scr2.py window just fine.

But, if I enter new selection dates in the two date fields on scr1.py
and click the submit-button again, I get this (a little edited):

Code:

 
  157    rdb2.read_only()
 
global rdb2 = <module 'rdb' from '/python_root/local/rdb/__init__.pyc'>, rdb2.read_only = <function read_only at 0xE0CE48>
 /python_root/local/rdb/__init__.py in read_only()
  100
 
  101 def read_only():
 
  102    Statement('set transaction read only').execute()
 
  103
 
  104 def read_write():
 
global Statement = <class 'rdb.Statement'>, ).execute = <method 'execute' of '_rdb.Statement' objects>

<class 'rdb._rdb_exceptions.Error'>: (-1015, '%SQL-F-BADPREPARE, Cannot use DESCRIBE or EXECUTE on a statement that is not prepared', '26000')
      args = (-1015, '%SQL-F-BADPREPARE, Cannot use DESCRIBE or EXECUTE on a statement that is not prepared', '26000')
      message = ''

As long as I do not click on any link in the result table (and run
the scr2.py script) I can enter new dates and rerun scr2.py any
number of times. Bu as soon as I have clicked on a "detail-link"
and runed the second script, scr1.py can't be re-runed again, I always
get the error above.

I have checked teh scripts and they both do the correct
transaction handling and so on, but this seems to be on
another (lower) level, so to speak.

It somehow self-heals since the PyRTE process seems to die
after the error and scr1.py runs OK after a few retries. But that
will not be accepted by the customer... Smile

Now, I *guess* this will run OK if each script runs in it's own PyRTE
process. What is the simples way to force each script to a separate
PyRTE process ?

My mapping looks like this (the relevant parts, as far as I understand):
Code:

...
set     /hv-bin/*                     script=as=ME
...

exec /hv-bin/*.py (cgi_exe:pyrte)/disk1/web/bin/*.py \
     script=syntax=unix script=query=none map=once ods=5
...


"ME" is the UAF user that the scripts are runed by.
It is not the english word for "moi"... Smile

I would be prepared to pay some for tips/pointers that gets
this going. A presentation for the customer is planned for
this Tuesday (in two days).
Back to top
View user's profile Send private message
jfp



Joined: 12 Jul 2004
Posts: 623

PostPosted: Mon May 10, 2010 8:53 am    Post subject: Reply with quote

Hi,

a few questions:
- Does scr2.py use a different database than scr1.py?
- If yes, do you use an alias for your databases?


JF.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
jfp



Joined: 12 Jul 2004
Posts: 623

PostPosted: Mon May 10, 2010 9:57 am    Post subject: Reply with quote

To be sure that there isn't any alias/filename problem you cn apply the following patch:

Code:
$ gdiff -p __init__.py
*** PYTHON_ROOT:[local.rdb]__init__.py;1        Tue Apr 27 09:07:42 2010
--- __init__.py Mon May 10 09:38:35 2010
*************** def attachDB(filename, alias = ""):
*** 123,128 ****
--- 123,130 ----
      global Databases, Statements
      alias = alias.upper()
      if Databases.has_key(alias):
+         if Databases[alias] != filename:
+             raise ProgrammingError
          return
      if alias == '':
          attachStr = "attach 'filename %s'" % (filename,)


JF
Back to top
View user's profile Send private message Send e-mail Visit poster's website
jescab



Joined: 28 Jan 2008
Posts: 254

PostPosted: Mon May 10, 2010 10:15 am    Post subject: Reply with quote

> > a few questions:
> - Does scr2.py use a different database than scr1.py?

No. Same DB and same table. Just different SELECT's (Group BY in
scr1 to get summary-records and all detail-records in scr2 "drill-down").
But still the same base records. And I have checked that I "rdb.commit()"
in both scripts. And I do not run them at the same time (but then,
WASD would have started a second PyRTE process, not ?).

> - If yes, do you use an alias for your databases?

No, just "rdb.dbattach('my-db').

I also tried to import the module rdb "...as rdb2" and prefix
all referensec with "rdb2." in one of the scripts, but that
didn't change anything.

I can not interpret the GDIFF output.
What should I change ??

Another thing...
I have (it comed from some example, I think) a "while"
at the start of the main python part that I do not understand what it does :

My main part of scr2.py (the one that crasches) looks like this :
Code:

while wasd.cgiplus_begin(True):

  rdb2.attachDB('MK000DB')

  if os.environ["REQUEST_METHOD"] == "POST":
    run_script ()
    print '</body></html>'
  else:
    print_html_form ()
    print '</body></html>'

So, I do not fully understand that part but I think that when I open
the page the first time, the "else:" part is runed and when I have
clicked the submit-button, the first ("then-part") is runned. But I do not
udnerstand how long it stays in the "while" loop...
Back to top
View user's profile Send private message
jfp



Joined: 12 Jul 2004
Posts: 623

PostPosted: Mon May 10, 2010 10:30 am    Post subject: Reply with quote

OK, if you use the same DB with no alias, I don't understand why you may have the -1015 error on the 'set transaction read only' statement.

the attachDB routine do nothing if you try to attach to the same database alias.

The patch just raise an error if you attach using a different database name but another database was already attached using the same alias.

From the pyrte.c source:
Quote:
The cgiplus_begin() method accepts zero, one or two positional parameters.

1) The first is a boolean (True or False) that if true allows a standard CGI
script to call this once before returning false at the second call. This
eliminates the need for explicit CGI and CGIplus paths.

2) The second parameter is also a boolean (True or False). If true it gets
the script code modification time to check for changes. If modified it returns
false causing the CGIplus script to exit gracefully (and consequently have the
new code run-up by the server).

NOTE: This RTE needs to differentiate between operating in CGIplus mode, where
the RTE activated scripts that are unaware they are operating in a persistent
engine environment, and when an explicitly CGIplus script is using the RTE as a
Python engine. Hence two flags in the code; IsCgiPlusMode which indicates it's
not standard CGI, just an RTE, and IsCgiPlusScript which indicates it's been
activated as a CGIplus script.



JF
Back to top
View user's profile Send private message Send e-mail Visit poster's website
bisonmalembouche



Joined: 01 Jun 2005
Posts: 74

PostPosted: Mon May 10, 2010 10:33 am    Post subject: Reply with quote

You should use the Python debugger to get more details about your problem
I see at
http://www.vmspython.org/DownloadAndInstallationPython

Winpdb is a platform independent GPL Python debugger with support for multiple threads, namespace modification, embedded debugging, encrypted communication and is up to 20 times faster than pdb

The doc is very good, see
http://winpdb.org/docs/
and
http://winpdb.org/docs/breakpoints/
http://winpdb.org/docs/exceptions/
Back to top
View user's profile Send private message Send e-mail
jfp



Joined: 12 Jul 2004
Posts: 623

PostPosted: Mon May 10, 2010 12:28 pm    Post subject: Reply with quote

About the script mapping,
You may try
Code:
..
set     /hv1-bin/*                     script=as=ME
set     /hv2-bin/*                     script=as=ME
...

exec /hv1-bin/*.py (cgi_exe:pyrte)/disk1/web/bin/*.py \
     script=syntax=unix script=query=none map=once ods=5
exec /hv2-bin/*.py (cgi_exe:pyrte)/disk1/web/bin/*.py \
     script=syntax=unix script=query=none map=once ods=5
...


Use hv1-bin for scr1.py and hv2-bin for scr2.py

I expect this will create 2 separate processes.

JF
Back to top
View user's profile Send private message Send e-mail Visit poster's website
jescab



Joined: 28 Jan 2008
Posts: 254

PostPosted: Mon May 10, 2010 5:53 pm    Post subject: Reply with quote

Just adding /hv2-bin/.. didn't change anything. It's still one single server process.

But when I also changed the "script-as" user on the second SET directive
(to another user that also can "read" the database), two server processes
was started and I can now switch between the windows without crashes.

That is, my current setup is like this :
Code:

set     /hv-bin/*                     script=as=ME
set     /hv2-bin/*                    script=as=MK

exec /hv-bin/*.py (cgi_exe:pyrte)/disk1/web/bin/*.py \
     script=syntax=unix script=query=none map=once ods=5

exec /hv2-bin/*.py (cgi_exe:pyrte)/disk1/web/bin/*.py \
     script=syntax=unix script=query=none map=once ods=5

MK is simply another username on the system that also have SELECT on the database.

The main application is runed frm the /hv-bin/ path and it
in turn calls the ssecond app through the /hv2-bin/ path.
Code:

00000446 HTTPd:80-3      HIB      4     5706   0 00:00:02.59      3653   2916 M
         [ME]                                                           23328Kb
00000447 HTTPd:80-4      HIB      4     5780   0 00:00:01.98      3050   2330 M
         [MK]                                                           18640Kb



This is good-enough for the demo on Tuesday, but I can not
create new users for every new application. I have to solve
the root cause of this.

I'd still liked to pay someone to help me with this. That is, someone
that known the WASD/Python/PyRTE enrivonments and that I can
proved a test-case that shows this errros. I can probably strip down
my scripts to something small that shows the problem.

Or should I (can I) post a test-case here ?

Finaly, let me say that the base technology works like a charm.
Each display of the detail info in the second browser-windows
draws aprox 0.01 CPU sec (DS20e @ 666 MHz) incl the SELECT
against the database and formatting of the HTML in the Python code.
It's hardly visible at all in MONITOR... Smile
Back to top
View user's profile Send private message
jfp



Joined: 12 Jul 2004
Posts: 623

PostPosted: Mon May 10, 2010 7:06 pm    Post subject: Reply with quote

Shouldn't be the rule an exec+ instead of an exec one ?
Not clear for me Confused

JF
Back to top
View user's profile Send private message Send e-mail Visit poster's website
jescab



Joined: 28 Jan 2008
Posts: 254

PostPosted: Mon May 10, 2010 7:11 pm    Post subject: Reply with quote

Well, if I knew the difference I might have been able to answer... Smile

Anyway, that is more of a WASD question then a Python one,
I think I'll make a pointer on the WASD mail list also. I think
there are many there that doesn't follow this thread...
Back to top
View user's profile Send private message
mark daniel



Joined: 14 Jul 2004
Posts: 10
Location: Adelaide, South Australia

PostPosted: Sat May 15, 2010 1:05 am    Post subject: Reply with quote

jescab wrote:
Well, if I knew the difference I might have been able to answer... Smile

Anyway, that is more of a WASD question then a Python one,
I think I'll make a pointer on the WASD mail list also. I think
there are many there that doesn't follow this thread...


The documentation (should) always use an exec+ but it doesn't make any real difference. The code sees the (rte)/blah/ construct as implicitly CGIplus.
_________________
A pox on the houses of all SPAMers.
Make that two poxes.
Back to top
View user's profile Send private message Visit poster's website
jescab



Joined: 28 Jan 2008
Posts: 254

PostPosted: Sat May 15, 2010 1:23 am    Post subject: Reply with quote

OK. So no difference there then. Fine.

One thing that I doesn't really understand right now is how the different .py files are "runed" by the PyRTE, so to speak.
Are they always runed "from start to end" at each call ?
Or are they in some mysterious way "hanging on" waiting for the next call (from the browser) ?

My view was/is something like this :

- Traditional CGI : The script is runed in a new process at each call, and always runed from start to end.
More or less like running "$ python some-script.py" from the VMS command line.

- FastGCI : The scripts "hangs-on" in the actual Python code and waits for the next call.
And that is what the "while wasd.cgiplus_begin(True):" loop is there for (?).
This can only use one specific PY file in each persistent process, right ?

- PyRTE : The Python interpreter (runtime envir) is kept alive, but each script runed by it runs more or less as in the trad-CGI case.
And in my view, a Python script runed by the PyRTE should not have that "while wasd.cgiplus_begin(True):" loop, not !?

In the actual case with which I have the problems reported, one of
the script does have the "while wasd.cgiplus_begin(True):" loop while the other doesn't.
That was a quick test to see if that might have any rellevanse to the problem at hand.
I haven't tried it with both scripts having the loop commented out, thought...

Anyway, the bottom line is that I do not understand how the PyRTE "works" and that is probably part of "the problem"... Smile
Back to top
View user's profile Send private message
mark daniel



Joined: 14 Jul 2004
Posts: 10
Location: Adelaide, South Australia

PostPosted: Sat May 15, 2010 1:43 am    Post subject: Reply with quote

jescab wrote:
Just adding /hv2-bin/.. didn't change anything. It's still one single server process.


This works with a standalone CGIplus application. The path to script is used as a differentiator. Different paths equals different scripts. However in this case it's an RTE that is to be engaged and WASD just finds the first free (suitable) one available. The script itself doesn't come into it. It's up to the RTE to do any subsequent processing.

jescab wrote:
But when I also changed the "script-as" user on the second SET directive
(to another user that also can "read" the database), two server processes
was started and I can now switch between the windows without crashes.


The '(suitable)' above includes account, so you will have separate processes (RTEs) if distinct accounts are required.

jescab wrote:
This is good-enough for the demo on Tuesday, but I can not
create new users for every new application. I have to solve
the root cause of this.


I am still to understand the problem exactly. It appears as if Rdb spits for some reason but is that an RTE issue, a Python/Rdb interface issue, or something else?

jescab wrote:
Finaly, let me say that the base technology works like a charm.
Each display of the detail info in the second browser-windows
draws aprox 0.01 CPU sec (DS20e @ 666 MHz) incl the SELECT
against the database and formatting of the HTML in the Python code.
It's hardly visible at all in MONITOR... Smile


RTEs are remarkably efficient in activating scripts. The actual cost is pretty-much determined by the underlying engine. Once instantiated this is often, as in the case of Python, quite affordable.
_________________
A pox on the houses of all SPAMers.
Make that two poxes.
Back to top
View user's profile Send private message Visit poster's website
jescab



Joined: 28 Jan 2008
Posts: 254

PostPosted: Sat May 15, 2010 2:03 am    Post subject: Reply with quote

> I am still to understand the problem exactly.

When running two different PY scripts, buth using some Rdb code, one
of them crashes as described earlier (*if* they sheare PyRTE process).

> It appears as if Rdb spits for some reason

Yes, it appears so. I do get an Rdb error. The Rdb code in the two PY scripts seems to interfere with each other.

> but is that an RTE issue, a Python/Rdb interface issue, or something else?

And *that* is the $10.000 question... Smile

Let me ask a fairly simple question.
Should I, or should I not, use the "while wasd.cgiplus_begin(True):" construct in the PY scripts that are to be runed by the PyRTE ?

> RTEs are remarkably efficient in activating scripts.

Yes, but in the CGIplus case, where each process always runs a single PY script (right ?), wouldn't the performance be mostly
the same as when using the PyRTE ? At the cost of more VMS processes depending in the number of different PY scripts used, of course.

My view was that the main point with the RTE is to have fewer processes that are capable of running multiple *different* scripts,
but otherwise performes more or less like CGIplus. And that is why I want to use the RTE, to save memory on our maxed out 4 Gb DS20e... Smile
That might of course be totaly wrong... Smile
Back to top
View user's profile Send private message
mark daniel



Joined: 14 Jul 2004
Posts: 10
Location: Adelaide, South Australia

PostPosted: Sat May 15, 2010 3:51 pm    Post subject: Reply with quote

jescab wrote:
> I am still to understand the problem exactly.

When running two different PY scripts, buth using some Rdb code, one
of them crashes as described earlier (*if* they sheare PyRTE process).

> It appears as if Rdb spits for some reason

Yes, it appears so. I do get an Rdb error. The Rdb code in the two PY scripts seems to interfere with each other.

> but is that an RTE issue, a Python/Rdb interface issue, or something else?

And *that* is the $10.000 question... Smile


Excellent. I was wondering what the value of the pay for help was to be.

Also good to see it's in dollars and not euros Very Happy

jescab wrote:
Let me ask a fairly simple question.
Should I, or should I not, use the "while wasd.cgiplus_begin(True):" construct in the PY scripts that are to be runed by the PyRTE ?


Only if the script is intended to be used in both CGIplus mode (script stays active inside persistent RTE) and in CGI mode (script starts and stops inside persistent RTE).

For CGI (non-plus) the construct while wasd.cgiplus_begin(True): might as well not be present. It returns true the first iteration and false the next (dropping through to exit).

The Python script (bytecode) is cached for efficiency, marginally reducing subsequent startup latency. The original file (Python source or bytecode) is checked before reactivation for changes in modification timestamp and reloaded if required.

jescab wrote:
> RTEs are remarkably efficient in activating scripts.

Yes, but in the CGIplus case, where each process always runs a single PY script (right ?), wouldn't the performance be mostly
the same as when using the PyRTE ? At the cost of more VMS processes depending in the number of different PY scripts used, of course.


Yes, provided the script itself doesn't have significant instantiation costs. If script startup (ignoring engine startup) costs are significant the CGIplus can be useful if that latency is an issue of the script is heavily used.

For a lightweight script both should have similar latencies and throughput under load.

jescab wrote:
My view was that the main point with the RTE is to have fewer processes that are capable of running multiple *different* scripts,
but otherwise performes more or less like CGIplus. And that is why I want to use the RTE, to save memory on our maxed out 4 Gb DS20e... Smile
That might of course be totaly wrong... Smile


Your understanding is exactly correct. You are looking for a permanently instantiated engine that handles multiple scripts.

I am currently looking at the PyRTE engine code for clues.
_________________
A pox on the houses of all SPAMers.
Make that two poxes.
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    forum.vmspython.org Forum Index -> Python for OpenVMS All times are GMT + 2 Hours
Goto page 1, 2, 3, 4  Next
Page 1 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group