I have a client that would like to export from the Central Repository using a scripted approach using al_engine to get specific objects. So far so good, that’s working OK. Now they want to export something other than the current version of an object. Is this possible?
eganjp 
(BOB member since 2007-09-12)
no, Central Repository operations are only possible from Designer, these options are not available from command line
manoj_d (BOB member since 2009-01-02)
Did you mean that these options are NOT available from the command line?
eganjp (BOB member since 2007-09-12)
yes, I mean NOT available from command line, edited that in the post
manoj_d (BOB member since 2009-01-02)
I had been prototyping a Continuous Integration solution on my local environment with a local repository, so just assumed that the command line approach of using al_engine to export dumps of the repo could be extended to the central repository. Actually, to tell the truth, I assumed there would be even more command line options available for the central repo, such as versioning.
From what I have read, it appears that this is not supported. The ideal scenario would be to automate the following:
By having this intermediate requirement of having to first download the central repository definition into a local repository it appears that true Continuous Integration is looking impossible to achieve.
Can anyone give any tips on how you might set up Continuous Integration with a Data Integrator project?
gje306 (BOB member since 2013-01-28)
I think you’re going to have to provide some context around what “Continuous Integration” is before we can provide any feedback.
eganjp (BOB member since 2007-09-12)
Hi Jim. In this context, the Continuous Integration Environment (CIE) exists to test that all the components in our product are working together correctly. The Data Services module is just one part of the overall product.
The ideal scenario is that the entire CIE is automatically rebuilt every night from the latest code base and then unit and integration tests executed against it, including the Data Services module.
I have so far been unable to find a way of automating an export of the latest Data Services central repository code.
gje306 (BOB member since 2013-01-28)
OK, I see what you are trying to do. However, just because an object is in the Central Repository doesn’t mean that it is ready for prime time. If you have developers that aren’t clued in to what it means to check an object into the Central Repository you could get code that is a partially developed state because it was time to go home and it was checked in without ever being unit tested or even being validated. Perhaps you have a more disciplined group of developers than I’ve seen at some client sites. 
eganjp (BOB member since 2007-09-12)
Ha! I just realized that I started this thread. 
eganjp (BOB member since 2007-09-12)
Jim
This is something I am also trying to do. Your original post states:
From that I assume you already have solved getting the latest version from the repository using a scripted approach - care to enlighten us on how you did it?
surreydude 
(BOB member since 2013-03-14)
al_engine -g -Q -NMicrosoft_SQL_Server -S -XXD@ExportTest.xml@@D
I’ve found that some versions of DS 12.x work better/differently than others. With 12.1.1.4 the above command will generate a more or less empty file. DS 12.2.2.2 will produce a file with the most recent version of the object (the syntax of the -XX paramter is slightly different in 12.2.2.2)
I have a theory that if you tweak the central repository table al_version and set repo_type = null and secure = 0 that the al_engine command might work using DS 12.1.1.4. Don’t forget to change repo_type and secure back to their previous values.
eganjp (BOB member since 2007-09-12)
Thanks - I thought as much; that’s what works for me
surreydude (BOB member since 2013-03-14)
To test out the al_engine command I performed a full repository export using both al_engine and then through data integrator. When comparing the two atl files (using beyond compare), they appeared to be completely different files, however they did have exactly the same size.
A quick reorder with the sort command e.g.
sort data_integrator_dump.atl > data_integrator_dump_sorted.atl
sort al_engine_dump.atl > al_engine_dump_sorted.atl
followed by a hash of both e.g.
md5sum data_integrator_dump_sorted.atl
12345 data_integrator_dump_sorted.atl
md5sum al_engine_dump_sorted.atl
12345 al_engine_dump_sorted.atl
revealed that they had an identical hash after sorting. This suggests a near enough 100% probability that these files are identical after ignoring the order of the elements of the file.
gje306 (BOB member since 2013-01-28)
I have since performed some further analysis to test whether or not the export from a central repo using al_engine is the same as an export from a local repo using Data Integrator following a get latest from the central repository. The local repository was created from scratch, so this was the initial get latest into the repository. I used Get Latest with filtering to include all components associated with the repository.
In the first case the command I used was:
al_engine -UMY_USER -PMY_PASSWORD -SMY_CONNECTION -NORACLE -X
Which according to the help docs should export the repo to a .atl file
-X : Export the repository to file “repo_export.atl”
In the second case to export from Data Integrator I selected the context menu from within a project view and selected repository–>export to file.
The project I have been analysing is a real world project 17MB in size. After normalising and sorting the content of both exports, there appears to be many many differences between them.
Importing the file exported using al_engine into Data Integrator and then exporting it, produces a third result again.
What I am trying to achieve is an automated way of extracting the latest version of the central repository. However, from this analysis, it appears that al_engine cannot be reliably used as a way of automating the current manual process of getting the latest version of the central repository ie. through Data Integrator and a local repository.
Can anyone shed any light on why we see these differences when exporting repositories in these different ways and whether there is a way to reliably automate the process of exporting from the central repo into a local repository?
gje306 (BOB member since 2013-01-28)
In some cases the differences are inconsequential. I’ve seen some odd stuff where using compare within Designer shows differences that while they don’t affect the execution of the job cannot be explained. Sometimes it is as simple as the exception catch shows a difference that doesn’t appear to be different at all.
You’ll have to provide specific examples of differences that you see.
Are you doing full exports? I don’t think I would want a full export. I would be much more comfortable with an export of the objects (or the jobs) from the Central Repository that are running in production. Why would you try to test code that isn’t in already production? I sure wouldn’t want to be running code that wasn’t even out of development.
eganjp (BOB member since 2007-09-12)
Have you seen cases where the differences are consequential?
gje306 (BOB member since 2013-01-28)
Not yet. But I’m not looking at things with the same breadth as you are.
eganjp (BOB member since 2007-09-12)
Hi Manoj,
Can you please confirm that get latest from central to a local using the command line is still NOT available?
Thanks.
–
David
dlaplant (BOB member since 2006-03-27)
No, this functionality is not yet available
manoj_d (BOB member since 2009-01-02)
Like Jim, we’ve been exporting the latest version of objects out of central repos using al_engine. This is in support of automated CI testing. Just because something isn’t supported doesn’t mean it’s not available. 
Related question: has anybody figured out a way to export a specific table object via al_engine? I see, as of SP6, looking at al_engine /?, that you can delete a table object, but am wondering if the same syntax might work for exporting? Going to try…
JeffPrenevost (BOB member since 2010-10-09)