In Houdini 9.5, the help server is based on the Python Paste project. This makes it more robust and easy to configure, but also means the previous experimental method for running a stand-alone central help server are out-of-date.

So, if there are any brave souls out there who want to help me test the procedure for the new way to do it, I'd appreciate it.

Part of the problem is the extreme variance of Python setups and Python knowledge out there, but if we don't start somewhere, it'll never get done, so I want to try outlining a procedure and see how far people can get with it. Your feedback is greatly appreciated. Hope that's OK.

• Starting with a clean Python, install setuptools.
  
  
• Once you have setuptools, use easy_install to install Paste, Paste Deply, Paste Script, and Pygments.
  
  EDIT: you also need to easy_install demjson, a JSON parsing library.
  
  
• Copy the “houdinihelp”, “whoosh”, and “wikiparser” packages from $HFS/houdini/scripts/python/ onto your Python path.
  
  NOTE the version of Genshi included with Houdini is specific version that avoids an issue that was introduced into Genshi that prevents the Houdini help system from working. Using the current version of Genshi from the Genshi website will not work. We are working with the Genshi developers to rectify this.
  
  
• Copy the “genshi” and “Genshi.egg-info” folders from $HFS/houdini/scripts/pythonlibs/ onto your Python path.
  
  
• Make sure you don't have another copy of any of these libraries overriding them on the Python path.
  
  
• The Houdini help folder is here: $HFS/houdini/help The search index folder is here: $HFS/houdini/config/Help/index You can copy these folders somewhere central if you want.
  
  
• Make a copy of the server configuration file in $HFS/houdini/config/Help/deploy.ini.
  
  
• Edit the config file so the “index =” line at the top points to the search index folder.
  
  
• Change the “docs =” line underneath to a comma-separated list of directories containing help.
  
  If you list multiple directories, the later ones will overlay the earlier ones. You can use this to keep an original copy of the help in one directory tree and put your own files in other and overlay them.
  
  Do not leave @nodes, @shelf, or @path in the list: they are only meaningful inside a running Houdini.
  
  
• In a shell with the proper python path set up and the “paster” tool on the shell path (should have been set up with you easy_install'ed the Paste packages), try this: paster serve <config.ini> (where <config.ini> is the path to your edited copy of the configuration file.
  
  Paste lets you set up server options such as the port number in the configuration file (see the "" section) or on the command line, e.g.: paster serve –port 8888 config.ini See the Paste documentation for more information.
  
  To set up a central server that other machines can access, you'll want to change the “host” option from 0.0.0.0 to the domain name of the server. E.g. if you're running the help server on internal.company.com, change the “host =” line in the config file to internal.company.com, or set the option on the command line, e.g.: paster serve –host internal.company.com –port 12345
  
  
• Try viewing the help in a browser.
  
  
• In Houdini 9.1, open the embedded browser and click the “Settings” tab. In Houdini 9.5 (only works in versions after build 180), open the embedded browser and click the screwdriver (settings) icon at the top. Set the “External help URL” to the URL of the central server.
  
  
  Big note 1: some things in Houdini that cause the help to open (e.g. pressing F1 in a tooltip) won't respect the external URL setting. This is a bug in Houdini we need to fix. The external URL setting will need to move out of the embedded browser and become a real Houdini preference.
  
  Big note 2: As you create your own documentation, you'll want to index it (this is necessary not only to make it searchable but for lots of other things, including showing up in the table of contents). I'll provide a script you can use to update the index ASAP (just need to clean up the script we use during our build process).
  
  Finally, apologies to everyone who's been interested in doing this and has been frustrated by the lack of progress. A bunch of things changed about how the help works between 9.1 and 9.5 that reset the process, and a bunch of resources were diverted to other things. Hopefully now we can move forward on it.

Here's the script we use for indexing. In a shell with the necessary libaries on the Python path, do:


python index.py configfile


…where configfile is the configuration INI file you copied and modified from $HFS/houdini/config/Help/deploy.ini. The script will get the location of the index directory and document directory(ies) from the config file (see the setup instructions above).

This file should will be included somewhere in the standard Houdini install in the future.


import os.path
from optparse import OptionParser
from ConfigParser import ConfigParser

from houdinihelp import providers, indexing


def command(options, args):
if not args:
raise ValueError(“You must give a config file”)

filename = args
if not os.path.exists(filename):
raise OSError(“File %r not found” % filename)

configparser = ConfigParser()
configparser.read(filename)
configparser._defaults.setdefault('here', os.path.dirname(os.path.abspath(filename)))
configparser._defaults.setdefault('__file__', os.path.abspath(filename))

dirs = configparser.get(“DEFAULT”, “docs”)
provider = providers.provider_from_string(dirs)

indexdir = configparser.get(“DEFAULT”, “index”)

print “dirs=”, dirs
print “indexdir=”, indexdir

indexing.index_docs(provider, indexdir,
clean = options.clean, optimize = options.optimize,
prefix = options.prefix,
stems = not options.nostems, spelling = not options.nospell)

if __name__ == ‘__main__’:
parser = OptionParser(usage = “%prog configfile”)
parser.add_option(“–clean”, “-c”,
dest=“clean”,
default=False,
help=“Re-index from scratch, instead of incrementally”,
action=“store_true”)

parser.add_option(“–optimize”, “-o”,
dest=“optimize”,
default=False,
help=“Optimize after an incremental index”,
action=“store_true”)

parser.add_option(“–no-stems”, “-S”,
dest=“nostems”,
default=False,
help=“Do not update the stem database”,
action=“store_true”)

parser.add_option(“–no-spell”, “-s”,
dest=“nospell”,
default=False,
help=“Do not update the spell check database”,
action=“store_true”)

parser.add_option(“–prefix”, “-p”,
dest=“prefix”,
help=“Only index pages starting with a certain path prefix. FOR DEBUG USE ONLY.”,
default=“/”)

command(*parser.parse_args())

Since i am on my holiday and it's too hot and sunny outside, let me be the first guinea pig…first i'll make a huge pot of coffee, think i'll need it.

system spec (software wise…):
python 2.4.4, gcc 4.1.2, running on gentoo

/edit
installed flup, paste cheetah, pastedeploy, pastescript, pygments. Copied whoosh, wikiparser and houdinihelp directories to /usr/lib64/python2.4/site-packages. Copied the genshi direcotries to /usr/lib64/python2.4. Copied the help and index directory to /opt/hhelp. Edited the deploy.ini and copied it to /opt/hhelp as well. Set PYTHONPATH to /usr/lib64/python2.4usr/lib64/python2.4/site-pacakges. PYTHONHOME is set to /usr (anything else will break my gentoo system).

Running paster serve /opt/hhelp/deploy.ini results in quite a few error messages on screen. I'll try on some other linux distro to see if it works there.
/edit

Hi Matt.

Tried a few other things but can't get past the errors on my gentoo system. Here they are:


Traceback (most recent call last):
File “/usr/bin/paster”, line 7, in ?
sys.exit(
File “/usr/lib64/python2.4/site-packages/paste/script/command.py”, line 79, in run
invoke(command, command_name, options, args)
File “/usr/lib64/python2.4/site-packages/paste/script/command.py”, line 118, in invoke
exit_code = runner.run(args)
File “/usr/lib64/python2.4/site-packages/paste/script/command.py”, line 213, in run
result = self.command()
File “/usr/lib64/python2.4/site-packages/paste/script/serve.py”, line 252, in command
relative_to=base, global_conf=vars)
File “/usr/lib64/python2.4/site-packages/paste/script/serve.py”, line 273, in loadapp
return loadapp(
File “/usr/lib64/python2.4/site-packages/paste/deploy/loadwsgi.py”, line 193, in loadapp
return loadobj(APP, uri, name=name, **kw)
File “/usr/lib64/python2.4/site-packages/paste/deploy/loadwsgi.py”, line 213, in loadobj
global_conf=global_conf)
File “/usr/lib64/python2.4/site-packages/paste/deploy/loadwsgi.py”, line 237, in loadcontext
global_conf=global_conf)
File “/usr/lib64/python2.4/site-packages/paste/deploy/loadwsgi.py”, line 267, in _loadconfig
return loader.get_context(object_type, name, global_conf)
File “/usr/lib64/python2.4/site-packages/paste/deploy/loadwsgi.py”, line 389, in get_context
global_additions=global_additions)
File “/usr/lib64/python2.4/site-packages/paste/deploy/loadwsgi.py”, line 480, in _pipeline_app_context
context.app_context = self.get_context(
File “/usr/lib64/python2.4/site-packages/paste/deploy/loadwsgi.py”, line 397, in get_context
section)
File “/usr/lib64/python2.4/site-packages/paste/deploy/loadwsgi.py”, line 439, in _context_from_explicit
value = import_string(found_expr)
File “/usr/lib64/python2.4/site-packages/paste/deploy/loadwsgi.py”, line 18, in import_string
return pkg_resources.EntryPoint.parse(“x=”+s).load(False)
File “/usr/lib64/python2.4/site-packages/pkg_resources.py”, line 1912, in load
entry = __import__(self.module_name, globals(),globals(), )
File “/usr/lib/python2.4/site-packages/houdinihelp/__init__.py”, line 3, in ?
from whoosh import index
File “/usr/lib/python2.4/site-packages/whoosh/index.py”, line 360
d = “DEL” if reader.is_deleted(docnum) else “ ”
^
SyntaxError: invalid syntax


I suppose it's more an issue of this system but i would appreciate if you could point me in some/any direction. Btw, the ^M's in whoosh/index.py seem a bit evilish…

Tomorrow i'll give it a try on fedora9 and on ubuntu.

Looks like the problem is b/c you're running it with python 2.4.

If you've got python 2.5 on your system, you should use that.

I'm running SUSE 10.2 64 bit(python 2.5) and I'm almost there. Instructions were good, except that for whatever reason ‘paster’ wasn't installed to a system path - possibly because the installer seems to lean towards /usr/local and my python isn't there.

Anyway, if that happens to anyone, just copy it from:

/usr/lib64/python/site-packages/PasteScript-1.6.3-py2.5.egg/EGG-INFO/scripts

to a path like /usr/bin or /usr/local/bin if you prefer. Adjust the above path to match your python installation and version of PasteScript, and remember to ‘chmod 777 paster’.

Anyway, so close - I get a fail when running ‘paster serve /opt/HoudiniDocs/config/Help/deploy.ini’:

File “/usr/lib64/python2.5/site-packages/houdinihelp/wsgi.py”, line 12, in <module>
import demjson as json
ImportError: No module named demjson


I tried commenting that out, then commenting the reference to it in streams.py …then….stopping and reverting back and figured it's best to post. This seemed like it might be a work in progress for custom docs that got included in the 9.5.181 distribution. Anyone have a thought/suggestion?

Cheers,

J.C.

Oops, sorry demjson is also a required library.

ops:

easy_install demjson

And it works! woohoo!

Need to start looking into custom stuff now…

Cheers,

J.C.

I'd HIGHLY recommend using something like virtualenv to install all these additional packages to, instead of to your system's python directories. It'll save you tons of headaches later.

I didn't find it that bad - I'm on top of everything I've done and can undo in a second. I would have stopped if anything collided with existing versions, though. Virtual envs are for wimps!

Ummm…headaches later? :?

Cheers,

J.C.

JColdrick
Ummm…headaches later? :?

Yeah, like if you install some package through your package manager that also depends on one of these python libraries you've install manually. Your package manager doesn't know anything about the stuff you've thrown in there, so it happily overwrites stuff, and you're left with some kind of mutant hybrid of two different versions.

Same thing goes for upgrades to system-managed software, or even removing a package. It may happily delete files that you need for your manually installed python packages.

And then Matt may decide to use a different version of Paste that conflicts with the previous version, and you want to have both available at the same time…then you've got a big headache.

JColdrick
Ummm…headaches later? :?

I got a huge migraine here…thought updating python here would be a no brainer but ran in to a little error. Solution is to delete the old version of python (v2.4) and then install the other. Funny thing is my package manager needs python…if i delete it i get some spectacular crashes…

Maybe i should install Vista… :roll:

I know gentoo makes it hard to have multiple versions (2.4/2.5) of python installed.

Debian and Ubuntu make it pretty easy, although I'm not sure about other distros.

If you think Vista would be easier…good luck

catlee
If you think Vista would be easier…good luck

Isn't it point and click then? I was informed that it was just that easy…


Anyway, runs fine on ubuntu 8.04 32 bits.

I'm thinking the ideal might be to create downloadable turnkey zip files for Windows and Mac/Linux that contain the libraries, docs files, etc., and a script that sets up the proper paths and start the server (as long as you have Python 2.5 ).

Yeah rub it in Matt, thanks. :evil:

Got it running on my p3-450 mhz running 32bit archlinux. Bit slow but it runs.

I'm thinking the ideal might be to create downloadable turnkey zip files for Windows and Mac/Linux that contain the libraries, docs files, etc., and a script that sets up the proper paths and start the server

Sounds like a good idea. Will this mean the help files won't be part of the houdini downloads anymore but a separate one? Would make a lot of sense for keeping the help server up to date, you just download the new doc package when you added some new info Matt.

Cheers,
Hans

Pagefan
Sounds like a good idea. Will this mean the help files won't be part of the houdini downloads anymore but a separate one? Would make a lot of sense for keeping the help server up to date, you just download the new doc package when you added some new info Matt.

I don't think it would be possible to take the docs out of the product – it would be too complicated for new users. I was thinking about a separate download that also included the docs and everything else you need for a central help server.

Forgive me for missing this but I can't find it on the forums - where do I find information about implementing custom local docs? I got the indexer but your comment seems to infer a working knowledge of incorporating custom docs into the Houdini tree…

Cheers,

J.C.

Well, I'll spew some information and you can tell me if this is what you were asking…

The help server has the ability to overlay the contents of multiple directory trees. In the default configuration ($HFS/houdini/config/deploy.ini) that includes all HOUDINIPATH/help directories (that's what the special “@path” token does in the “docs =” line of the config file).

So, if you want to add a page at the root level of the help server, you could e.g. create “$HOME/houdiniX.X/help/mypage.txt”. Instructions on the wiki format the docs use is in the “How to use the help” chapter of the help. You could then view your page in the embedded browser at http://localhost:48626/mypage.html. [localhost]

The tricky part is building the index. Since the configuration file that ships with Houdini depends on being run from inside Houdini (or at least hython – it imports hou for several things like getting the houdini path and pulling help out of digital assets), you'll probably need to get the stand-alone thing working before you try re-indexing using the script above.

In the list of directories on the “docs =” line of the configuration, later entries override earlier ones (the opposite of the Houdini path :twisted: ). So if e.g. you've got a standalone server setup where the original docs are in “/A” and your docs are in “/B” (and your config file says “docs = /A, /B”), you can override an original doc under “/A” by recreating the same path under “/B”, e.g. “/B/nodes/sop/copy.txt”.

Does that help?

Enough to get me into trouble! Thanks!

Cheers,

J.C.