MVS/380 MVSSRC code
Brought to you by:
kerravon86
Menu
▾
▴
Tree [58ed5f] master / History
kerravon86
Read Me
MVS source code reconstruction project -------------------------------------- Concept ------- Imagine an imaginary project called MVS/380 1.0 and imagine that development of it is underway, and that you can find the source code for this project in MVSSRC.V1R0M0.MANASM and it is still incomplete (not getting past the testing/QA team). The QA team has identified a list of problems, which can (briefly) be found in MVSSRC.V1R0M0.DIFF. There are a plethora of tools and listings also available to assist with the "debugging process", but simply looking in the DIFF dataset will show the state of progress (at least for modules that have been given to the QA team to date). This MVS/380 1.0 project is massive, and expected to take 10 years to complete. In the meantime, life goes on, so there are other "unrelated" projects underway (MVS/380 1.1, 1.2, 2.0 etc), that mostly don't actually require source code, so are being developed differently (working at the binary level). That isn't entirely accurate, but it explains why the very old MVS/380 1.0 datasets are still shipped, and most work is done on that project. Introduction ------------ MVS 3.8j was the last free version of MVS shipped by IBM. Source code was nominally available, but the machine-readable version was out-of-date, and the only other thing available was listings via microfiche, but no guarantees they were up-to-date either. The only thing we can rely on is the load modules themselves. Everyone is basically happy with these load modules, so any source code effort needs to be based around that end-product. A further complication is that there are varying levels of maintenance applied to the load modules, plus various usermods. So there's not even a single "MVS 3.8j". A theoretical version of MVS 3.8j that had every single bit of maintenance that IBM put out may be logically impossible due to conflicts, wasn't shipped in that form anyway, and is not very usable due to things like the lack of 3390 disk support. So while there is logic behind wanting to obtain source code for a "pure MVS 3.8j" system for historical reference, it is also logical to want to obtain it for every 3.8j-based distribution. Hence the focus here is on the MVS/380 1.0 distribution. Getting the source code up-to-date for it will automatically ripple through to subsequent versions of MVS/380 1.0 and will more-or-less cover TK3 also. Auto Project ------------ This is a theoretical project, which may start at any time, to build MVS/380 1.0 from assembler source code that was produced via disassembly of the existing modules. This can be done automatically, although without the base registers identified, the code will be really horrible. Having said that, someone may spend the minimal effort required to identify base registers, and the code gets better and better. That project, if it existed, would have source code in a concatenation like this: //ASM DD DSN=MVSSRC.V1R0M0.AUTOASM,DISP=SHR // DD DSN=MVSSRC.V1R0M0.MANASM,DISP=SHR What that means is, as soon as the original source code in MANASM is updated, there is no logical purpose for the AUTOASM version, so it is deleted, exposing the manual version in the concatenation. This means that in the source code repository, the file is deleted from the auto-1_0 branch. What this means is that when a source file is initially imported into the repository, it gets an auto-1_0 branch, in reflection of the fact that the manual-1_0 is not up to date. But once the manual-1_0 is brought up-to-date, it is deleted. Since the auto project doesn't actually exist yet, except in name, in practice this means that the auto version is deleted before ever having any code committed on it. In addition, in practice it is not usual for anyone to want to use the auto code for that purpose. Theoretically it would be good to have proof that the assembly and link process are all working fine, producing the required executables. In practice, no-one wants to go to the effort of doing that, when the executables are the end product and already working. The exception is when people want to change those executables and need access to the source code. But again in practice, the sort of people with the skills and desire to make such code changes also have the ability to get the source code in synch in the first place, so that is what they do. In addition, there are volunteers who will bring source code up-to-date themselves if they know someone else wants to work on it. However, the placeholder is there. And what that means is that if there is any change, other than total deletion, between tk3 and auto-1_0, then someone has put the auto version on the head (perhaps to fix an important bug that just required a change to a constant), and thus any automatic updating of either the head, or releases between 1.0 and the head, should come from the auto branch rather than the manual branch. Arrangement ----------- Code is only imported when there has been some activity on it, e.g. a load module is now being built from source, or simply that disassembly activity has started. This will probably change later, and the rest of the code imported. When a file is imported, projects are automatically set up for it. Of most interest is the project manual-1_0 This is a manual effort to reconstruct MVS/380 1.0 (which is roughly the same as TK3 which is roughly the same as MVS 3.8j). The initial seed for manual-1_0 is the source code found on TK3, which is roughly at the MVS 3.7 level. It is assumed (even if it is known to be untrue) at the outset that the manual-1_0 branch is not up-to-date, so an auto-1_0 is created as another branch off it. That is a separate project, described later. At the same time that the file is added to CVS, it is also added to a control file (or virtually added, at any rate), called control.txt, which says whether the head version (ie which signifies a future version of MVS/380), should be automatically updated from new versions on the "auto" branch. Note that an auto file will be an exact copy of the manual file unless the auto version doesn't have an up-to-date manual version, in which case a diassembly will be used instead. It is assumed the answer to that is "yes", so an entry only needs to be added if the head is a replacement, and the old historical code is of no interest. So adding: iebgenrt repl signifies that the module iebgenrt has been replaced and no automatic updating should be done. Under this scheme, people are always free to edit the head version of the code, e.g. to change a delay timer, without needing to bring the manual-1_0 up-to-date first (although they are of course encouraged to do so). The principle is that it is better to get that timer fixed rather than insist that a whole lot of ancilliary work be done before the useful fix can be accepted. It also means that people are free to make incremental improvements on manual-1_0 without needing to do the whole thing, nor be involved in future versions of MVS/380. The manual version will not necessarily assemble successfully. Although once it is up-to-date, it is expected to remain working. Basically people are free to do whatever they are interested in, and that effort is captured for whoever needs it. Tags ---- It is assumed that the latest manual-1_0-v tag has already been applied to the head. If new files have been added to the repository since then, they will not have that tag. Updates to the manual-1_0 branch are not automatically applied to any other branch. Other branches have their branch point marked with something like this: release-1_3-v4 release-1_3-v3 release-1_3-man-v27 release-1_3-v2 release-1_3-v1 So that someone maintaining the 1.3 release (when e.g. we are up to release 1.5 already) can choose to merge or not. Obtaining --------- Here is what you need to do to check out the different versions if you wish to submit an update: If you want to view/change the manual version of the code (this is the source that was supplied by IBM, but is often out-of-date), then do this: cvs -z3 -d:pserver:anonymous@mvs380.cvs.sourceforge.net:/cvsroot/mvs380 checkout -P -r manual-1_0 mvssrc If you have your own sourceforge id, you can do this instead: cvs -d :ext:kerravon86@mvs380.cvs.sf.net:/cvsroot/mvs380 get -r manual-1_0 mvssrc If you don't have CVS, then you can browse the code online here: http://mvs380.cvs.sourceforge.net/viewvc/*checkout*/mvs380/mvssrc/asm/iebgen03.asm?pathrev=manual-1_0 to look at individual files, in this case, iebgen03.asm If you are instead interested in the disassembly, then replace the manual-1_0 above with auto-1_0, as such: cvs -z3 -d:pserver:anonymous@mvs380.cvs.sourceforge.net:/cvsroot/mvs380 checkout -P -r auto-1_0 mvssrc cvs -d :ext:kerravon86@mvs380.cvs.sf.net:/cvsroot/mvs380 get -r auto-1_0 mvssrc http://mvs380.cvs.sourceforge.net/viewvc/*checkout*/mvs380/mvssrc/asm/iebgen03.asm?pathrev=auto-1_0 And finally, if you wish to enhance MVS, choose from: cvs -z3 -d:pserver:anonymous@mvs380.cvs.sourceforge.net:/cvsroot/mvs380 checkout -P mvssrc cvs -d :ext:kerravon86@mvs380.cvs.sf.net:/cvsroot/mvs380 get mvssrc http://mvs380.cvs.sourceforge.net/viewvc/*checkout*/mvs380/mvssrc/asm/iebgen03.asm The latest version of these instructions can be found here: http://mvs380.cvs.sourceforge.net/viewvc/*checkout*/mvs380/mvssrc/readme.txt Macros ------ The sys1.maclib that is distributed with MVS/380 1.0 is NOT what was used to build the MVS/380 1.0 executables. Ditto for all other distributions. The maclib that was used was probably something between 3.7 and 3.8j. The maclib was probably not even given a version number. As part of this project, we also need to reconstruct that maclib. Failure to construct it means that some things don't even assemble, and even if they do, they will be different from the load module, which takes away our ability to automatically verify the source is up-to-date. The assumed interim maclib has now been given a name - backmac v500000. It is hoped that the bulk of the MVS code has been compiled against this to-be-constructed maclib. If some code is found to have been built against a more recent maclib than that, it will be given a name such as v700000. An older version would be v300000. It is hoped that this will cover the bulk of the problem although in the worst case scenario, a single source file may have been compiled multiple times against different maclibs, and via patching, even the same source file could be using different versions of the macro in the same compilation. A more-complicated scheme will be set up if and when required. Note that the OS/360 MVT 21.8f distribution is a good source of backlevel macros, but normally they are too old. Given that these old maclibs are still in use, and given their importance, they will be distributed as part of MVS/380 in the future, until such time that they are no longer in use. ie there may be several versions of backlevel macros. Disk space is not considered to be an issue, and the files aren't that big anyway (8 MB each). A similar situation will apply to SYS1.MODGEN - backgen. You can obtain all the macros of interest as follows: Backmac v500000: cvs -z3 -d:pserver:anonymous@mvs380.cvs.sourceforge.net:/cvsroot/mvs380 checkout -r backmac-v500000 mvs380/source/maclib OS/360: cvs -z3 -d:pserver:anonymous@mvs380.cvs.sourceforge.net:/cvsroot/mvs380 checkout -r os360 mvs380/source/maclib MVS/380 1.0: cvs -z3 -d:pserver:anonymous@mvs380.cvs.sourceforge.net:/cvsroot/mvs380 checkout -r release-1_0 mvs380/source/maclib Names ----- Default names for everthing will be MVSSRC. MVSSRC.MANASM MVSSRC.BACKMAC.V500000 MVSSRC.AUTOASM MVSSRC.JCL MVSSRC.DISDATA MVSSRC.CNTL MVSSRC.ASM In general the files will reflect the latest status of the latest operating system. The source control system (not even on MVS) has no problem keeping track of the multiple versions of things. The "unwieldy" MANASM - manual assembly is like that to reflect the fact that it is usually not up-to-date, and so you can't just hop into it and start modifying it. There is a MVSSRC.CNTL(STATUS) that shows the status of the current modules. If a module has matching MANASM and AUTOASM, then it will be marked "match". If the ASM has caused a complete replacement, it will be marked "repl". If you wish to make modifications to a future version of MVS/380, you have the following options: 1. The simplest option is to just update MVSSRC.ASM and make your modification. Any executable that was newly built, was built using this version of the code. However, this version of the code is normally a very horrible-looking disassembly, and unless you are making a trivial change, like changing the word "STOP" to "HALT" because you saw a typo in an error message in one of the modules, you cannot do this. The reason you cannot do this is because code can't be inserted unless you have all the offsets as labels instead of offsets. However, if you are willing to construct a patch in a format that doesn't disturb any existing offsets, then you may do this. The text change above is an example of something that doesn't change any offsets. The principle here is that we want your bug fix. 2. A much better option is to update the DISDATA etc to make the autoasm version look much nicer, by letting it know what base registers are in use etc so that sensible labels can be created. When it looks reasonable, you send in the new DISDATA etc, plus the auto-generated assembler from that, *plus* your modification. This will then make it into a future ASM. Note that according to Gerhard, it only takes a few minutes to get the assembler looking nice, so if it takes you more than that to do, you know who to contact. 3. The ideal option is if you can get the MANASM up-to-date. Then submit the up-to-date MANASM *plus* your changes to it. Note that you shouldn't automatically mark your new version as a replacement. Maybe your up-to-date MANASM has made its way into some TK3-based project, and someone has then changed that to use a constant name instead of a hardcoded x'80', or any other change that would be welcome in MVS/380. 4. At any time you can choose to ignore the existing assembler and write a complete replacement, or perhaps use one of the others as a base, but radically change it such that you want the new version to be used going forward, and never subject to replacement. This would then be marked as "repl" and never subject to alteration from other projects. A point of clarification is in order. The "ASM" version may contain things like comments such as "This variable needs to say HALT rather than STOP to be compatible with z/OS". These comments will never be seen in the autoasm version, because the comments are not in the load module, even though the word "HALT" ends up in the load module. The principle here is that we don't want to lose those comments either (or more likely, a more complex change, perhaps a macro, which the disassembler does not recognize because no-one has yet made the effort to get that recognized). Tools ----- The main focus is to get the manasm up-to-date. As such, there are "listing" files available to assist in this process. MVSSRC.MANASM.DISLIST MVSSRC.AUTOASM.DISLIST These are the result of running two disassemblies on the corresponding load modules, with some manipulation to get it into a machine-friendly format. MVSSRC.MANASM.DISLIST2 MVSSRC.AUTOASM.DISLIST2 These have had the offsets and logical line number stripped, so that they don't show up in diffs. MVSSRC.DISLIST2.DIFF A diff of the above two sources. MVSSRC.MANASM.DISLIST3 MVSSRC.AUTOASM.DISLIST3 These have had operands and object code stripped, so that there is just the instruction name on its own, to prevent still-incorrect labels from showing up as a difference. MVSSRC.DISLIST3.DIFF A diff of the above two sources. This is the suggested place to start the process. This diff shows the first logical line where differences start to occur. You can then take a note of those line numbers and go back to AUTOASM.DISLIST2 (not 3) and cut and paste the exact lines (even though they have hardcoded offsets, and not suitable for long term use) into an appropriate spot in manasm (you presumably will have taken a backup of the file you are working on before you start working, so that you have that as a reference). You can find an appropriate spot in manasm by getting the offset as shown in DISLIST for the starting line number, and then finding that same offset in MANASM.LISTNOG, and then getting the logical line number from that (note that this necessitates that PRINT NOGEN be in effect), at which point you know where to insert any new lines of code (and comment out any ones no longer used). This manipulation of the code normally requires reference to any macro expansion, so you probably want to take a look at MANASM.LISTGEN to get a listing with "print gen" in effect. If the instructions that are changing are in fact part of a macro expansion, then you don't need to insert the code, you need to instead find (or create) the proper macro that this code was compiled against, and put it into BACKMAC.V500000. Theoretically the above process may be possible to automate, with a failsafe of refusing to insert any code that is a victim of a wrong macro (finding a suitable macro replacement could theoretically be automated though, if there are several to choose from). Generally you probably just want to insert the first block of changes, and get feedback that that has taken effect correctly, before moving on to the next block. Otherwise you may have a rogue ANOP thrown in as a change. When a DS area is encountered, things become a bit tricky. On the one hand you want any new instructions in that position to be inserted. But this may be junk rather than new instructions. To start with, the DS area will be used to get the disassembler to ignore that, so you won't see a difference. If there were in fact instructions inserted there, and now they've just been masked out, this should become evident later when you try to get the offsets lining up correctly and you find you need to put some padding in. Use a DC for the padding and the instructions will show up, and you can insert the new instructions instead. If there is no evidence of the new instructions (or data) being referenced, then this may in fact be a DS area, so you can change the DC to an unnamed DS. When there are no differences at the instruction or data level, you can now go back to the beginning and this time look for any operand differences. These should now be showing up in MVSSRC.DISLIST2.DIFF. From here you get the line number. You go to that line number in the LISTNOG file. Actually you open that LISTNOG twice. On one screen you look at the line number with the problem. On the other you scroll to offset difference suggested by DISLIST2.DIFF, and look for a label there. The most likely cause of this is a variable being moved around, so try moving the variable to the suggested spot, or more likely inserting a dummy variable to get it positioned correctly. Now there should be nothing showing up in DISLIST2.DIFF. So, it's time to change the hardcoded offsets (from the earlier cut and paste) to point to real labels. The same technique above applies with the 2 screens looking at LISTNOG. Have a look at the offset listed, and there is probably a label already there. So make the operand (e.g. a branch destination) point to that label, and see if that works. Note that all changes made to the code should be marked as FIXME!!! until such time as they have been finalized and hopefully reviewed. Note that a lot of the earlier work is somewhat mechanical and doesn't require any real programming knowledge, but until a programmer has analyzed the code change for macro recognition and hardcoded constants and commenting, it needs markers to warn anyone who may change this code in the future that it hasn't been properly inspected yet (even though it is generating perfect code as-is). These files have listings from the three different sources: MVSSRC.MANASM.LISTNOG MVSSRC.MANASM.LISTGEN MVSSRC.AUTOASM.LIST MVSSRC.ASM.LIST Note that the DS detection is switched off when AUTOASM is being run outside of this activity, because the MANASM is not a reliable place to get DS info from until it is up-to-date. However, if MANASM is seen to be up-to-date with DS detection switched on, then it will be accepted as up-to-date and replace the AUTOASM entirely (and that change will propagate into ASM too). JCL --- Each source file has an associated (and automatically-generated) JCL file that performs a standard build for use in development. It does the following (catering for the general case): Assemble manasm into temporary object and create MANASM.LIST Run parser to extract DS offsets Run initobj to zero out DS data (step may be optional) Link with standard object code to build executable it is part of, into temporary load module Disassemble that module into DISLIST, using storage markers Generate DISLIST2 Generate DISLIST3 (these two steps are required because the disassembler doesn't generate DS X'00' and instead generates DS X'12...') Run disasm on real executable into temporary dataset, using storage (DS) markers Run assemble on it again, and link a temporary executable (end of annoying steps) Run disasm on it again, storing result into AUTOASM.STORLIST Do a diff between MANASM.DISLIST and AUTOASM.STORLIST to create STORLIST.DIFF (this should be empty when the job is done) (the following steps are really only included for consistency, the results shouldn't change) Run disasm on real executable, without DS markers, into AUTOASM.DISLIST The object code is permanently saved. Generate AUTOASM.DISLIST2 & 3 (end of consistency steps) Create diffs of the two dislist, dislist2, dislist3 files which are all useful during the development phase, but even when the job is finished, dislist will still show differences due to the DS sections which show up as garbage constants in the real executable (but will be zero, one way or another, in our executables). This may seem convoluted, but it's simply an automatic generation of all the files that will be of interest, so that the developer has all the info he needs at his fingertips, whenever he makes a change to manasm.asm. BUILDING -------- To create the MVS datasets, the following steps are done: getmacs - gets all the backlevel macros from the relevant branches allmvs - pretty much does everything else to set up the mainframe doone - useful for running a single file from the PC autoadd - used to add a new source file (see documentation in file)
