Archive

Archive for February, 2012

Support Details

February 24, 2012 Leave a comment

Someone recently showed me http://supportdetails.com/ and I was so impressed by it.  What a simple thing to set up and make available to everyone.  The “Email” feature is perfect for a SaaS provider who needs more information about the client side details (if they don’t have it built into their service already).

Our software keeps track of browser name/version and IP address for our users but knowing things like screen resolution, browser size, and Flash version would be an added benefit when tracking down those pesky browser specific problems.

I’ll also plug the IE6 Countdown since I’m very excited for IE6 to be a thing of the past.  Just think of all the code that will disappear and man hours that will be regained by not having to account for IE6.

Advertisements
Categories: Uncategorized

DataBee and DataMasker

February 11, 2012 Leave a comment

Being our DevOps guy I get to be involved in a variety of projects.  I was recently tasked with rebuilding our demo database to bring it up to date with our production system.  Our production database is ~15T but we only want a subset of that data for demonstration purposes.  The task is to pick a client or two and export a referentially correct set of data from production and import into a demo database. Once we have the data, it needs to be scrubbed so there are no traceable demographic information or client specific details.

Side Note: We previously performed this task using Oracle’s DataPump utility (EXPDP and IMPDP) to export Schema + Data based on a configuration file (INCLUDE, EXLUDE, QUERY rules).  That worked pretty well but was very time consuming to generate the conf file and run the export/import.  This approach requires DBA level access to both source and target databases.

What is Database Subsetting?

It is the process of creating a referentially correct cut down “subset” from a larger database.  Ideal for creating test, development, or demo databases from a much larger staging or production instance.

Options

I was hoping the open source Jailer project would work but it would have been very impractical with a database of our size.

I then came across www.grid-tools.com and downloaded a demo version.  I quickly found out that all of the features I wanted to evaluate were all disabled in their demo version.  I stopped there since it was going to involve sales people and live demos just to get a full version I could play with.

Then I came across databee.com and I was hooked.  DataBee offered a full featured, 30 day trial of their software.  You have to request an evaluation license but that’s acceptable to me.  I was contacted by a rep from the company who was 1) very friendly, 2) straightforward with pricing, 3) genuinely wanted to help, and 4) not pushy about the sale.

The software was a quick install (Windows only) and has a great interface and is very intuitive (at least to a technical person).   I ended up doing a live demo with them a couple days later and was even more impressed by the software.  The part I love is how easy it is to look under the hood.  Log files are readily available, all the PL/SQL code that is being executed can be found with a click of a button, and they provide a number of tools that show you exactly how your database relationships are setup, and how to generate the appropriate rules.

Once I got DataBee down and was able to create a subset of data (over Oracle database links.. yay!), I wanted to try masking/scrubbing this data.  The rep pointed out DataMasker to me and I went through a demo on that as well.  It was so easy to identify schemas, tables, and columns using a set of rules that would substitute and then synchronize data between tables.  You can even run arbitrary PL/SQL code as a masking rule.  They provide around 60 different data sets: First/Last Names, Date Variance, SSN, Phone numbers, Address, City, State, Zip, etc.

Database Subsetting is non-trivial but the DataBee product makes it look easy.  I highly value a software vendor who is honestly willing to help and not be pushy when it comes to the sales side of things.

One last point.  The thing that hooked me was how responsive the reps were to our needs.  I used DataBee on a table of ours with non-standard column names (lower case and Oracle reserved works — bad idea I know, it’s a legacy table).  I was able to grab the exact code causing the problem, submitted it to them with a description of what was happening, and they sent me a new install with the problem fixed in about 24 hours.  You know a software company is doing something right when they can patch and ship in  a short amount of time.

Categories: Uncategorized Tags:

Things I Learned in January

February 1, 2012 Leave a comment

January

  1. Ruby’s IRB can help in more situations than you realize.
  2. Egglectic in Wheaton, IL is very good.
  3. Giordano’s Chicago Pizza is awesome.
  4. Empty
  5. Empty
  6. Default MTU size for Ethernet v2 = 1500 bytes (28 header + 1478 payload).  Send >1478 bytes and it will not be delivered.
  7. Empty
  8. Empty
  9. Bash alias is executed when defined, functions are executed at runtime (Learned this in December but count it here).
  10. Oracle: Local schema objects can interfere with schemas of the same name.
  11. Increased knowledge of WordPress themes (outside of work).
  12. SQL*Ldr supports a FILLER keyword to skip columns.
  13. Oracle’s query result cache. Link
    1. SELECT * FROM v$result_cache_objects;
    2. SELECT * FROM v$result_cache_statistics;
    3. BEGIN DBMS_RESULT_CACHE.FLUSH; END; /
  14. USA Hockey Registration is done annually, valid from Sept 1 to Aug 31 of the following year.
    1. Blogger Templates (blogspot.com) are one big crazy XML document that include HTML, CSS, and Javascript.
  15. 3 Rules of Logic
    1. “Whatever is – is” – Law of Identity
    2. “Nothing can both be and not be.” – Law of Exclusion
    3. “Everything must either be or not be.” – Law of excluded middle
  16. 1,000,000,000 nano-seconds = 1 second.  MS Active Directory stores some values as 100-nanosecond intervals since January 1, 1601 (UTC).
  17. xvfb – X virtual framebuffer
    1. iceweasel is a firefox port for debian
    2. xvfb + v11vnc + vncviewer = pretty sweet
      1. /usr/bin/Xvfb :99 -screen 0 1280x1024x24 -ac
      2. x11vnc -display :99
      3. vncviewer host:port
      4. export DISPLAY=:99 && firefox/iceweasel
    3. Watch out for Oracle package state during a unit test suite. Constants defined in one test can wreak havoc in others.
    4. http_proxy environment variable can mess up gem installation: gem install [name] –source gems.myserver.com
  18. Oracal DataPump: INCLUDE, EXCLUDE, QUERY.  EXPDP/IMPDP can do schema or schema + data.
  19. ssh -X user@server – Enables X11 forwarding
    1. Introduced to http://en.wikipedia.org/wiki/Levenshtein_distance
    2. http://www.wolframalpha.com
  20. ssh-copy-id user@dest-server (To cut out the password prompt and use your ssh key instead)
  21. Hoagie Haven in Princeton, NJ was a dream come true.
  22. Deep Fried French toast at Hightstown Diner in New Jersey was amazing and the 6 egg omelette was easy to eat.
  23. Javascript Lesson:
    1. !! is a way to convert a reference or value to an actual boolean value.
      Unnecessary if you aren't explicitly checking against that type.
      The following progression is effectively what happens when everything evaluates
      
      !!$('realElement')
      !(!elementReference)
      (!false)
      (true)
      
      !!$('notRealElement')
      !(!null)
      (!false)
      (true)
    2. Learned 4 new board games: Ticket to Ride: Nordic Countries, Can’t Stop, King of Tokyo, Airlines Europe
  24. Learned how to play the card game Dominion.
    1. The Prudential Center in Newark is where the New Jersey Devils play.
    2. Newark is not a safe place to be.
  25. GMail requires the Return-path header property to be set in order to receive mail.
    1. I later found out this was fluke. GMail really cares about a valid SPF DNS record for your domain.
  26. Oracle will segfault when compiling objects if a synonym dependency is invalid.  Recreate the synonyms (even if they appear to be working) and then recompile the objects.
    1. Encountered this error message for the first time when running: select * from <table>
      ORA-14137: Table in partially dropped state, submit DROP TABLE PURGE
      14137. 00000 -  "Table in partially dropped state, submit DROP TABLE PURGE"
      *Cause:    An attempt was made to access a partitioned table in a partially
                 dropped state.
      *Action:   Submit DROP TABLE  PURGE to drop the table.
  27. Learned how to use DataBee.com’s  Set Designer, Extractor, and Loader.
    1. 2.5 hour lesson on Life Insurance, there’s some neat options out there.
    2. Introduced to me http://xebia.github.com/Xebium/
    3. git commit –ammend
    4. Gerrit + Jenkins
  28. Zdeno Chara (NHL) can shoot a 108 MPH slapshot!
  29. The Relativists’ Dilemma: Forfeit your worldview or your humanity.
    1. Can a relativist accuse others of wrong doing?
    2. Can a relativist make a charge of unfairness or unjustice?
    3. Can a relativist improve their morality?
  30. Vasili Arkhipov
  31. http://www.phpbench.com/