a DirectX importer addon for Blender 2.6

first goals :

. to import anything from an .x file.
  obviously verts, faces but uv, armatures, weights, normals...
. import .x in binary format too

horizon :
. export to .x or mod/co-maintain the existing x exporter.
. this project is also a prototype for a 'Blender Exchange Layer' project.
  BEL would be a common layer logically located between an importer/exporter
  addon and the blender data format, that would allow :
    . to provide a common set of methods to retrieve/inject objects in Blender
    . to provide a common set of transformation and selection tools between an
      import/export script and Blender datas (rotate, rescale, filters...)
    . to provide a common set of parsing helpers for new io addons

PLY won't be used unfortunately (it's way too slow as far as I tested)


TO TEST THE SCRIPT :
  . copy the 'io_directx_bel' in /scripts/addons or addons_contrib
  . start blender
  . enable then addon in  user prefs > addons
  . run the script with file > import > directX
 
25/01/12 0.18
  . code rewrite according to api bmesh changes (me.polygon and uv layers essentially)
  . implemented foreachset() call for uv import (faster)
  
25/01/12 0.17
  . faster, 60% faster in some case : various loops improvements, infile templates parsing disabled by default
    saw another bottleneck about data chunks as string but will wait for binary support for better point of view.
  . interface cosmetics
    
23/01/12 rc 0.16
. committed to svn (and littleneo git as usual)
. corrected a bug about referenced token parenting
. corrected a bug about non parented meshes
. armatures/empties importation enabled by default
. last run importer options are saved in a 'last_run' preset,
  so it can be replayed or saved under another name once a
  particular .x profile has been defined
. tagged script for 2.6.1

12/01/12 rc 0.15 :)
. name conversion changed from 5 to 4 digits
. matname, imagename and texturename fixes :
. a path test is made at image import time with any existing data images, 
  so a same file cant be loaded twice wathever the naming method, / or \, rel or abs etc 
  (bel.material.new() and after line 835)
. image and texture names should be ok now (tested with : incrediblylongname.x)
. materials are replaced accordingly in existing objs when using the 'replace' naming method
. fyi, the Dx exporter has the following inconveniences :
    . split linked faces into individual faces
    . inversed uvmapping (y axis) ?
    -> see testfiles/blender_x_export/incrediblylongname.x
   
29 and 31/12/11
. Cosmetics, code cleaning and optimizations
. bpy.ops.object.select_name methods replaced with
    ob.select = True
    bpy.context.scene.objects.active = ob
. corrected a big bug about tokens info appending in dXtree()
. new bpyname() method in bel module. removed bel.common

26/12/11
. armature import and bone max. length option

23/11/11
. contrib candidate :)
. solved some naming cases, added bel methods.
. added experimental option about parenting (no armature yet just empties, when needed)
. a bit faster
. added some test files (empty parenting, armature etc)

22/11/11
campbell feedback (cont):
. added naming methods as options (default is blender name inc. if name exists)
  me and ob remove() should be ok with special cases (empties, mesh with multi-users)
. improved ui


21/11/11
campbell feedback :
. converted immutables to tuples : templates_x.py and some other vars.
  http://stackoverflow.com/questions/3340539/why-tuple-is-faster-than-list
. dprint() (console debug) removed, replaced by a inloop tests (a bit faster)
  I'd like to keep it for now for easier debug (eg user feedbacks with 'processing' option)
  
19/11/11
. object parenting support. parsing the x tree from roots using import_dxtree()
  actually faster than before
  
16/11/11
. weight group import
. improved ui a bit and console logs
. x matrices to blender ones conversion

14/11/11
. global matrix options
. added messy code about binary (not working)

11/11/11
. import materials and textures (basics) : uv and image mapped (multitex mode)
  and material created with tex slot if any. alpha should be ok.. ?
. added a smooth options
. better tolerance with faulty .x (upper/lower case of template names)
. token names length from x to blender conversion should be ok also (long name cases)
. corrected a parser pointer error after one array parsing case.
. added more templates (for mat and tex support)
. removed texture files from repo in testfile (tex does not match meshes )
  added some other x files for further tests in binary and compressed format
  ( http://assimp.svn.sourceforge.net/viewvc/assimp/trunk/test/models/X/ )
  
08/11/11
. turned into an addon (fork from obj import so unused functions atm)
  enable it in addon, then file > import > directx
. splitted directx parser (io_directx_bel folder) and bel draft 
  the bel folder is intended to be located in /scripts/modules (shared components)
  but it's ok in scripts/addons too (tbd)
  bel folder (will) includes anything related to blender data helper (read/write)
. corrected duplicated quotes for x string type

07/11/11
. uv import
. generic directx token parser. templates items are used to read datas of any token type
  a bit slower but cool since it should support non strict standard directx files
  virtually it can retrieve everything from now supposing the template is know
  by default or given in the file. calls are now like :
		nbslots, mats = readToken('MeshMaterialList001') or
		uv = readToken('uv001') or
		nVerts, verts, nFaces, faces = readToken('Hydralisk_backbone1') etc
. removed the specific mesh parser the 'rigid' file is the last before mutation to
  generic parser. a bit faster but harder to make evolve or adapt. keep it as a faster
  strict 'branch'
. added some default templates
  goals / wip :
  . to compare template declaration in file and default one.
    so either use the default one (strict) or the .x one (could differ)
  . use by the generic data parser to avoid a parser for each kind of token
. cleaner code (grouping methods, function names, docs etc) 
  functions separated from calls
  renamed token dict as tokens etc
. added tweaks at the beginning of the file :
	chunksize = 1024     # size of file streams red in a row
	quickmode = False    # this to only find meshes (no parenting, no other tokens than Mesh ones)
	showtree = False     # display the entire token tree in the console
	showtemplate = True  # display template datas found in file
. added a patch for malformed datas of vertices (meshFaces) :
	# patch for malformed datas not completely clear yet ( I guess
	# there's bunch of us when looking at meshface syntax in .x files) :
	# when array members are like 3;v0,v1,v2;,
	# and not like 3;v0;v1;v2;, depending on template declarations.
	# http://paulbourke.net/dataformats/directx/#xfilefrm_Use_of_commas
. the script now generates linked faces (was not my fault !)
  > it seems Dx always separate each face :
  so it defines (vert * linked faces) verts for one needed vert
  the readvertices loop now remove duplicates at source
  it uses a verts lookup list to redirect vert id defined in faces

06/11/11
. vertices and faces imported from each test files
. added some info to test yourself in README 
. switched to binary for .x as text to retrieve eol (pointer bugs). should be ok whatever it's win, mac or unix text format,
  also works with mixed eol.
  it seems python 3.1 can't return a 'line' when data.realine() when read mode is 'rb' (U default and universal ? really ? ;) ) 
  when file has mac eol (\r)
  -> read(1024) in binary, decode, and replace any \r with \n. yes, it doubles lines for windows and lines value is wrong for now
  -> but the used pointer value is always ok now whatever the file format and still way faster than a data.tell()
  see CRCF folder to compare output wispwind.x by format.
. files are still splitted into chunks (1024 B) and readable as lines
. references : added 'user' fields when token is used. users store a reference with their childs but with a '*' tag at chr0.
  the tree reflects the changes
. now read anything and add it to the 'tree'. this includes unknow tokens.
. references are recognized. by reference I mean fields like { cube0 } rather than an inline frame cube0 {
  declaration.
  I don't know if one item can be referenced several time or referenced before declaration
  should be.. waiting for a case. for now only one 'parent' token, messages will show up
  multi references to one token if cases arise. 
. more permissive syntax : 'frame spam{', 'frame     spam   egg{', 'frame spam egg  {'..
. comments are recognized (inlines ones not done yet, since still no useful data red :) )
. header is red
. found other .x test files here :
  http://www.xbdev.net/3dformats/x/xfileformat.php
  created from 3ds max
. added .x files in repo. line 70 and following to switch.
. some token comes with no names, add a noname<00000> to them
. console gives line number (more useful than char position I guess)


05/11/11	day 0 :

. made some disapointing test with ply (from a speed point of view, else it looks really cool)
. made my own parser
. nothing imported for now, it's more about self-eduction to .x and concept
. but it reads the .x structure and can gather some info

resource gathered :

http://paulbourke.net/dataformats/directx/
http://www.informikon.com/various/the-simplest-skeletal-animation-possible.html
http://msdn.microsoft.com/en-us/library/windows/desktop/bb173011%28v=VS.85%29.aspx
http://www.toymaker.info/Games/index.html



step 1 : read main structure :

    read main token names (any 'template', any 'frame', any 'mesh')
    stores names in a token directory :
        token['template'] for templates :
            token['template'][templatename]
            token['template'][templatename]['pointer']          (int) chr position in .x file (tell() like*)
            token['template'][templatename]['line']             (int) line number in .x file
        token['frame'] for frame and mesh type :
            token['template'][frame or mesh name]
            token['template'][frame or mesh name]['pointer']    (int) chr position in .x file (tell() like*)
            token['template'][frame or mesh name]['line']       (int) line number in .x file
            token['template'][frame or mesh name]['type']       (str) 'ob/bone' or 'mesh'
            token['template'][frame or mesh name]['parent']     (str) frame parent of current item
            token['template'][frame or mesh name]['childs']     (str list) list of child frame or mesh names
            token['template'][frame or mesh name]['matrix']     (int) for now chr position of FrameTransformMatrix

at the end of step 1 the script prints a tree of these datas

step 2 : read template definitions :

    for each template in dict, populate definitions in it.
    it creates new fields in each token['template'][templatename]
    according to values found in .x :
        token['template'][templatename]['uuid']                 (str) <universally unique identifier>
        token['template'][templatename]['members']['name']      (str) member name
        token['template'][templatename]['members']['type']      (str) DWORD,FLOAT etc keywords or template name
        token['template'][templatename]['restriction']          (str) 'open' , 'closed' , or the specidied (restricted) value

that's all for now.

idea would be to allow 2 steps importation and random access to file :

    . first the file is quickly parsed. we only retrieve main info, nothing about verts, faces etc
    info like number of mats, textures, objects/mesh/bone trees
    for now : 150000 lines in 5 secs for step 1
    . then user select what to import
    . then the script retrieve selected datas according to selection, using the 'pointer' value
      to seek() to the needed data, then grab/parse/translate in something usable.
    . template are used at this point to know how to parse a specific part (adaptive parser)
	  
    so far this looks fast.
	
tested on windows. can be important because of eol and the code I wrote to compute pointer value.
(data.tell() is slow)
only one .x file tested, header is : xof 0303txt 0032 (windows \r\n eol)

don't know a lot about .x format :

uuid : 
  are the member/restriction always the same for a same uuid/template ?
  template name can vary for a same uuid ?
syntax :
  blank lines IN a stream of a {} section, after ; ?
  comments // and # IN a stream of data ?
  '{' and '<something>' and '}' on the same line or '{' '}' are always unique ?