Application to export POD documentation from SVN/GIT to Foswiki portal

Introduction

Leader.IT uses a Foswiki Web portal to manage documentations, also this document is written with Foswiki.
Unfortunately, the current environment does not allow an easy publication of written documentation in the source code.

The source code is written in different programming languages: perl, javascript, sql, bash.
Moreover these languages are plugged in and mixed with other languages like html, xml, css, json and template languages like DTL and Mason.

We have mixture of OOP, procedural, markup, templating language; so we have to use a common language to manage source code documentation.

Leader.IT source code is managed in SVN repository with the same users of Foswiki portal. Then is possible to update the documentation using the same user who performed the commits in SVN server, so the revision of the document will be aligned.

In Foswiki the documents are organized in Web (or Subweb) and Topics. More details at TopicsAndWebs

Specification

• The application scripts will be developed with Perl and eventually in Bash
• The application must be compatible with Subversion and GIT using Perl package VCI - A library for interacting with version-control systems.
• Updates will be launched from crontab and/or SVN hook and/or GIT hook
• The changes are assessed by asking the SVN/GIT server and remembering previous release imported
• It is possible to define initial version to import, then all following version are imported with their respective authors, so in Foswiki these revision are saved with different releases and authors like SVN/GIT repositories
• Documentation is writted with POD syntax
• In languages different from Perl, the POD text is included in comment sections using the specific language syntax (eg. /* ... */ in Javascript)
• The extractor has to handle POD comments in all the languages mentioned in this document.
• Each file in SVN or GIT repositories that contains POD are exported to Foswiki using a configuration file to know the correspondence between the repository resource (URI) and the Foswiki Web (or Subweb) destination (eg. https://www.leader.it/svn/masonsql/masonsql_dojo1.9/htdocs/welcome.html ==> http://www.leader.it/MasonSQL/SourceCodeHtdocsWelcomeHtml)
  • Specifies what part of the URI matches the Web or Subweb destination (eg. https://www.leader.it/svn/masonsql/masonsql_dojo1.9 ==> http://www.leader.it/MasonSQL)
  • Specifies the initial release to import
  • Specifies the suffix of destination Topic (eg. SourceCode)
  • The remaining URI (eg. htdocs/welcome.html) is converted with UCfirst in every portions of names and removing separators ("/" ".") (eg. SourceCodeHtdocsWelcomeHtml)
  • The config file format is TXT file
    • A row starting with '#' it is a comment
    • Fields are separated by blanks or tab
    • The fields are in order: Type (svn|git), Start release, Source base URL, Dest base URL, Topic suffix
• POD is converted to Foswiki syntax using package Pod::Simple::Wiki with twiki format
• POD links are converted to Foswiki links (using the rules in configuration file)
• It is possible to use Foswiki syntax in POD fragments (please specify limitations and precautions for a correct interpretation)
• New doc is saved using the same user who performed the commits in SVN server
• Each Topic have parent linked to a Topiccorresponding to the parent folder
  • There is a special file named as parent directory and with suffix .pod than it contains POD exported in Foswiki with the name of the parent directory
  • On bottom of folder Topic there is a list with children links and descriptions
  • Description is arranged with the NAME POD section
• To create files and dirs topics should be used Foswiki templates
  • %STARTSECTION{type="POD"}% define where to insert the POD document
  • %STARTSECTION{type="PODlist"}% define where to insert the children list (only in parent template)

Some explanations

• Initial and subsequent run have not differences. In subsequent run if the file is already loaded, you must import only the new releases; with new files, you must import releases with release >= Start release
• If a file not contains POD, it is not imported.
• Every release >= Start release in SVN/GIT will correspond to a release in Foswiki.

Implementation

• Program flow:
• Populate the /opt/svngit2foswiki/var/foswiki.cfg file as per all the modules which will be checked out with svn. In case you miss some the file created after the run missingfos.cfg will give you hints to add more entries into the foswiki.cfg file.
• Crontabsvnpodfos.sh runs from user www-data cron at 2300 every night. It does the following steps
  1. Exports all svn source files with the command svn export https://www.leader.it/svn/testpod and or other modules https://www.leader.it/svn/masonsql into /opt/svngit2foswiki/var/svn/ directory
  2. After export it calls the ppod.pl script from the directory /opt/svngit2foswiki/bin which does the following
     1. ./ppod.pl /opt/svngit2foswiki/var/svn/ > ../var/out
     2. For each file containing pod the program will
     3. Match the filepath against the foswiki.cfg directory path ( shortlist the matching files )
     4. For each file will generate corresponding output pod file to check if output pod exists
     5. Check for last updated version in the foswiki.cfg file and compare to the svn info Revision and shortlist the files
     6. Obtain all svn log revision dates and check against the foswiki date obtained from command view -raw all /Trash/TestPOD/AmitTest11.
     7. Pick files having svn log revision date > foswiki last date
     8. Produces the output files ppod_output.sh containing the foswiki save commands for selected files containing pod, it can be scanned to check the program flow and check for any discrepancies.
     9. All the intermediate files ( eg. previous versions of each file ) will be checked out into the var/FOSWIKI/svn/ directory and separate directories for previous releases and the output pod files
  3. The shell script svnpodfos.sh then calls the ppod_output.sh for Foswiki Portal pod file and timestamp updation with the command ". /var/lib/foswiki/bin/ppod_output.sh"
  4. It runs from www-data user in cron, for other users it could be run by the following steps
     1. sudo rootsh
     2. sudo -i -u www-data bash
  5. BUG: Overwrite foswiki.cfg file with updated nfoswiki.cfg for the next run ( containing updated Revision numbers ). This should not be done since this updated last revision number should not be compared as there could be multiple files with different last revisions within the same directory in foswiki.cfg file, and the svn log dates are being compared instead with the foswiki timestamp.
• Testing done
• Intermediate directories being produced ok in the FOSWIKI/svn/ directory for previous versions and output steps in output out file.
• All the commands for previous versions save should be in order in the runout.sh file with the correct corresponding timestamps from svn
• Check that all files containing pod only are taken into account and ones without pod are ignored
• Test for wsvn/ in the config file
• Further modications
• svn info ( procedure get_svnrevision ) can be removed as we are now using getallprevversions by xml log ( and no real need to svn info on each file )
• Revisions are still being compared in one or two places and will have to be removed later ( now comparing svn log dates with foswiki METAINFO timestamp )
• svn diff could be done for new revisions of files containing pod ( presently all new versions are sent for foswiki save which doesnt take any action if the pod information remains identical to the previous version )

Useful links

• Foswiki WelcomeGuest
• Foswiki System.TwentyMinuteTutorial#A_6._Create_a_new_topic_and_edit_an_existing_one
• https://en.wikipedia.org/wiki/Software_documentation#Technical_documentation_embedded_in_source_code
• CommandAndCGIScripts Foswiki CGI and Command Line Scripts
• svn export https://www.leader.it/svn/testpod *