Northern Grit http://www.northerngrit.com/feed en-us http://blogs.law.harvard.edu/tech/rss Sweetcron ash@northerngrit.com The Database Countryside Code: Best Practices for BI & SQL Users http://www.northerngrit.com/items/view/3006 Those ‘City Folk’ among you may not be aware but in Rural England we have what is called The Countryside Code, it’s a set of guidelines that everyone should follow in order to keep the countryside clean, tidy and a nice place to visit.  You may be asking – what does this have to do with Business Intelligence and Database Administration?  Well, I think it’s vital – if we all follow a fairly simple but broad set of guidelines then all classes of database user will have a better experience from Developers to DBAs and Analysts to CIOs.  This isn’t really about making your databases perform better, it’s about working better with each-other and taking other people’s perspectives on board.  Having been in most of the related roles over the years this is what I’d put into The Database Countryside Code… 1. Enjoy the countryside and respect its life and work Whether your application is an ‘out of the box’ software suite, a Business Intelligence package that can be tweaked on implementation or a hand-crafted bespoke solution if you’re running against a database maintained by someone else or shared with other applications you need to take heed of this point.  Remember that cooperation is key and if you build a good relationship with the DBA and the other key users of the database you’ll have a much better time of things and if there are any critical issues you’ll be included in the remediation process and may even be able to help your own users get back online faster.  It’s easy to see DBAs as grouchy, narrowly focused sorts who tend to view all user activity as bothersome (I can say that as I’ve been one myself) but generally speaking if the DBA is aware of user activity at all the chances are that there’s already a problem as it’s the long running, resource intensive activity that will stand out in alerts and performance reports.  Before your application goes live you should do some testing, run your designs and SQL statements / stored procedures past the DBA for some advice (but remember, you don’t have to take it) and establish some sort of procedure for reporting issues, and remember that an SLA can work both ways as you may need the DBA’s help as much as they might need yours.

  2. Guard against all risk of fire Security is a huge issue and as exploit frameworks and toolkits become more and more prevalent and feature-rich the likelihood of vulnerabilities being discovered in our applications should be treated more like a certainty.  If you’re developing bespoke applications and especially web apps you’ll need to pay close attention to the OWASP Top 10 application security risks but from a database perspective the most notable threat is SQL Injection - the art of passing SQL into an application so that it might be executed by the database (as a good starting point check out OWASP’s SQL Injection Prevention Cheat Sheet).  If you’re deploying packaged apps or BI tools don’t think that you’ve gotten away with it, the primary responsibility may be on software developers to avoid exploits but if they’re baked into an application you’re implementing it will affect your users and your business, so…

  3. Protect wildlife, plants and trees The most important security contribution we as implementers can bring to the table is to review and limit the privileges required by our applications.  Many install guides and expensive external consultants ask for a ‘dbo’ (database owner) level user and some even ask for ‘sa’ (system administrator) or ‘root’ level privileges but don’t hand these out like candy on halloween.  In most cases these high-level privileges are only required during setup and install and can be removed afterwards but often basic read/write access is all that is required (and for BI tools often read-only), it may only be achievable through a few frustrating rounds of trial and error but if you assign your applications the lowest possible permissions you will significantly reduce the risk of compromise in the future.  Another important step during implementation is to make sure that your permissions are segregated, where possible have a separate user for each service and an entirely separate user for accessing each database not shared by any other application.  Whilst it may seem excessive this setup will allow you to audit any security issues and identify which user was compromised and exactly what they had access to.

  4. Fasten all gates Many Business Intelligence tools include some degree of control over connection management and if you’re developing your own application you’ll have complete control over all database connections, the decision to be made is whether connections are ‘pinned’ open, closed after x minutes or closed at the end of each transaction.  The preference will vary depending on the load and the usage, in most Business Intelligence use cases there tend to be a large number of users, not always connecting concurrently and issuing fairly large queries against the database followed by periods of quiet whilst a report is read – in this case there is usually no need to keep the connection open for long.  On the other hand if you have users issuing a constant stream of small transactions (e.g. a Point of Sale system) the overhead of creating and dropping connections might actually add load to the database so it would be more effective in this scenario to maintain the connection.

  5. Keep your dogs under close control This applies more to developers and BI architects where your dogs are your users, if you are deploying an application that creates load on somebody else’s database you should do whatever you can to limit each user’s ability to cause long running queries – in some BI tools you are handed an option to let a query time out after x minutes and perhaps limiting the number of rows returned.  If you are developing your own application you should include both of these options but make sure that you kill the query at the database level rather than just killing the thread in your application that made the request otherwise it’s equally bad if not worse since the user may simply re-issue the offending query.  The actual limits are bound to vary from database to database but that’s where the first point comes in, discuss this with both your users and the DBA.

  6. Keep to public paths across farmland / Use gates and stiles to cross fences, hedges and walls When it comes to solving problems try to stick within the basic and simple boundaries of an ordinary user, avoid using undocumented stored procedures, excessive use of user defined functions, custom data types, plugins and extended stored procedures or anything else that strays too far from a standard install of the database platform.  Obviously you’ve got an app to deploy and you want to solve your problems in whatever way is best for your users but the further you are from a standard deployment the more issues you’re likely to encounter, both you and the DBA might be fully aware of this amazing new setting you tweaked to make things run better but a couple of years down the line during a disaster recovery will it all come flooding back quite as easily?  What if one or both of you that setup the application have moved on to other roles?  Thinking outside the box is great but be conscious of introducing risk and if you do feel that it is necessary then make sure that it’s well documented in the Run Book or the corporate wiki.

  7. Leave livestock, crops and machinery alone Since you may already have elevated privileges on your own database, a shared database or even the server you may be tempted from time to time to perform maintenance tasks or make minor ‘improvements’ to indexes or configuration settings – do not do so without the DBA’s blessing.  If you’re following the rules above you’ll probably have a fairly good rapport with the DBA already so it’s likely that you’ll be granted some level of trust not to mess things up but be careful not to overreach, the DBA will be ‘in the loop’ of many changes and other requirements (e.g. critical deadlines, disaster recovery tests, unplanned maintenance) whereas you may not be aware of them so before you make any changes run them past the DBA – just in case.

  8. Take your litter home / Help to keep all water clean If you’ve ever been a DBA you’ll have seen, on more than one occasion, tables popping up called tmpSomethingorOther, tblToBeDeleted or TableName_bak but when it comes to the key questions (How long have these been around?  Are they still required?) nobody seems to have a straight answer.  I know myself that whilst I’ve been developing data warehouses I’ve created these sorts of tables and subsequently forgotten what they were used for, not too much of a problem if you’re ‘the guy’ but in a large team or with personnel changes over time it can be hard to know what is required and what isn’t – I came to a database once with temporary tables over five years old which had not been deleted out of fear that they were important.  The moral here is an obvious one, clean up after yourself or if the table must exist for some short period of time put a note in your diary to come back and cull it.

  9. Make no unnecessary noise Be mindful of what errors you raise and what you write to public logs, if your application causes a large amount of data to be written to database or other centrally collated logs you may inadvertently make it harder to detect genuine issues which will hurt both you and and other users of the database.  If you do occasionally need exhaustive logs consider adding a ‘debug mode’ into your application which can be turned on or off via a configuration setting, that way you can turn it on whilst you’re tracing a fault and need more verbose logging then turn it off when you’re done.

  10. Take special care on country roads There can be plenty of unexpected hazards on country roads so don’t always rush around everywhere at 60mph, acknowledge that whist you might want everything to go as fast as possible you could be causing some other critical process to slow or stop.  Driving at night can be treacherous too as you might come across an unexpected backup window or import/export process, talk to your DBA and coordinate the major tasks.  If it’s a shared server make sure you have access to the task list so that you know where to slot in your jobs and that those jobs get put back into the master list.

Really it comes down to one thing, as the great and wise Jerry Springer oft said, “take care of yourselves, and each other”.

]]>
Fri, 30 Dec 2011 14:01:00 +0100 http://feedproxy.google.com/~r/bisql/~3/zDO19o98KOo/
Mobile Web Design Best Practices, Tips and Tricks http://www.northerngrit.com/items/view/3005 Everyone knows the key mantra for designing mobile web sites – “keep it simple” but there are some tips and tricks that will help to create a great user experience for mobile visitors…

Capture mobile users from the full site – if your full site isn’t rendering well on mobile devices how are people going to find the link to your mobile site?  Put in place a redirect to a mobile optimised layout though it’s worth remembering that redirects could also be annoying to users that wanted to see your main site so… Provide a link back to your full site – this could be in the footer or as a landing page but in some cases the user may be trying to achieve something not possible on a slimmed-down mobile site or they may be on a tablet that is incorrectly being recognised as a mobile device. Remember the bad old days - there are still a large number of mobile devices out there that do not fully support CSS and JavaScript, including older Blackberry models which are common in corporate environments.  If non-smartphone users are a target audience for your site it should be designed with these older phones in mind and progressively enhanced to support more modern design features and input validation. Consider multiple mobile layouts - you could have a theme that optimises content specifically for iPhone and Android, leaving the other mobile users with a plainer but still small-screen optimised site.  Figure out what your audience is likely to be using and target that but don’t forget to tweak and customise the site after you’ve gone live based on the type of devices your users are actually using which will change over time. Use appropriate input types – if you are asking the user to provide email address or usernames via a form it can be difficult for them to enter correctly if autocomplete is turned on, similarly it would be better to provide the numeric keypad if you are asking for a telephone number and you usually would not want .  You can provide this functionality with a mix of the <input> tag and the autocapitalize property, there are a whole host of other possibilities including length checking and regular expressions but bear in mind not every device will respect these features. Avoid scrolling – pagination vs. scrolling has long been a debate in web design circles but if you want to provide your users with a more ‘app-like’ experience the key elements to your site should fit adequately on the page without the need for scrolling.  This may not apply to content but if the user is being asked to follow through a process or provide a series of inputs it will be much clearer to the user what they have to do if it fits on one page, equally… Avoid clutter – if you have pages with little content it may be worth ensuring that any non-essential (but for whatever reason required) footer information sits below the bottom of the screen to avoid clutter, at the very least you should consider a little trailing white space followed by a dividing line to clearly separate the content from the footer. Consider the user’s goal – you might be falling over yourself to provide content or services to your mobile users but is that what they really want?  Consider whether or not the user might have other goals in visiting your site and show how they can be achieved, even if that is not via your mobile site.  For example, it may be helpful to include a ‘contact us’ or a telephone/email link on at least the first page if not every page. Don’t be annoying – it’s the little things that tend to irritate users and on a mobile device this is magnified since they are already compromising on screen size and input capability.  For example, pre-fillling forms with help text may mean that the user is going to have to delete that text to enter their own – irritating enough on a desktop and even more so on a mobile device. Device testing is essential – there are dozens of emulators and simulators for mobile devices but nothing will ever match testing on devices, it is very tempting as a developer to test primarily on a desktop but it really isn’t the same as holding a small device at arm’s length and using a tiny keyboard to provide input.  During your testing phase have someone with a very critical eye run through your site to check for any minor irritations, make sure to tell them to be ruthless in their criticism.

I hope that provides some useful information to those of you starting out with the mobile web and of course, much of this is up for debate so do get in touch if you disagree or have content to add.  The list is not intended to be exhaustive and over the next few months I’ll add posts on testing and more technical aspects of the process.

]]>
Wed, 21 Dec 2011 08:51:00 +0100 http://feedproxy.google.com/~r/bisql/~3/XuwJ-CxbY10/
How to bring OSX Lion’s Natural Scrolling to Windows 7 http://www.northerngrit.com/items/view/3004 If, like me, you move between Windows and Mac on a daily basis you may have found yourself finding it a little hard to figure out which way to scroll the mouse.  With OS-X Lion Apple introduced ‘natural’ scrolling which means that when you scroll the wheel on the mouse an upwards push sends the scroll bar down, that might sound weird but in essence your upward movement of the wheel actually pushes the screen upwards – very much like a touch gesture on a smartphone or tablet. Whether you love it out loath it, getting used to switching between the two is difficult and you could either turn it off on the Mac or if you like it you could bring the same feature to Windows.  As it happens the feature is already there, to enable it you need to edit a registry key and if you’re not familiar with this process I would advise caution since a mistake in the Registry can make your machine quite unstable but if you’re comfortable with RegEdit you’ll need to modify the following key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\HID\????\????\Device Parameters\FlipFlopWheel Set the value from 0 (default) to 1 where the ????\???? section are whatever device IDs you can see.  I changed the FlipFlopWheel property for all of the devices I could see, unplugged and re-plugged the mouse and the then it worked – natural scrolling on Windows. Credits go to darkfader on the NeoSmart forums for the original solution.

]]>
Tue, 06 Dec 2011 21:27:00 +0100 http://feedproxy.google.com/~r/bisql/~3/p9T6zgG0Trc/
London 2600 Xmas Party - Friday 2nd December 2011 http://www.northerngrit.com/items/view/2953 The traditional London 2600 Xmas Party will be on Friday 2nd December 2011.

Time from about 7pm onwards

Location: the upstairs room of the Nell of Old Drury pub.

For more details, suggestions, offers of help, offers of sponsorship etc:

Subscribe to the London 2600 email discussion list or email this blog via meetings@london2600.org.uk

]]>
Fri, 02 Dec 2011 16:04:00 +0100 https://p10.secure.hostingprod.com/@spyblog.org.uk/ssl/meetings/2011/12/london-2600-xmas-party---friday-2nd-december-2011.html
Quick Tip – PostgreSQL Equivalent of ISNUMERIC() http://www.northerngrit.com/items/view/3003 Very much like my previous MySQL ISNUMERIC() post I have recently been setting up a data source to collect records with telephone numbers from a Postgres database and one of the essential validation tests is to make sure that the field really does contain a number. Despite the fact that many regard Postgres as the best open source database platform I find myself frustrated by it’s lack of standard functions.  I understand that Postgres is designed to be extensible and that user defined functions can be built but I need my code to be both portable and read-only so I have to work with what I’m given.  Ideally what I’d be looking for is an equivalent of Microsoft SQL Server’s ISNUMERIC() or Excel’s ISNUMBER() functions but very much like MySQL I had to turn to regular expressions although as you’ll see, Postgres does not have a clean and clear REGEXP() function… SELECT DISTINCT contact_number FROM customers WHERE (contact_number ~ ‘^[0-9]+$’) I hope that helps any of you out there that encounter the same problem.  

]]>
Thu, 01 Dec 2011 10:03:00 +0100 http://feedproxy.google.com/~r/bisql/~3/97caXn_pLrU/
Apple Bans Security Researcher Charlie Miller For Exposing iOS Exploit http://www.northerngrit.com/items/view/3001 The latest wave in the infosec world is that Apple has banned the well known security researcher – Charlie Miller – from it’s developer program for exposing a new iOS exploit. It’s not really the smartest move as I’m pretty sure anyone as smart as Charlie Miller still has plenty of options – use another [...]

Read the full post at darknet.org.uk

]]>
Wed, 09 Nov 2011 13:44:00 +0100 http://feedproxy.google.com/~r/darknethackers/~3/8nZZCRvFZ64/
Apple shouldn't bother with TV... http://www.northerngrit.com/items/view/3000 If the telly ain't broke, don't fix it Analysis  Can nobody rid of us the barefoot CEO? He may be gone, but Steve Jobs continues to manipulate the press from the beyond – this time through his biographer, Walter Isaacson. The Steve Jobs biography launches the hype for Apple's next great product, a TV.…

]]>
Tue, 25 Oct 2011 13:35:00 +0200 http://go.theregister.com/feed/www.theregister.co.uk/2011/10/25/apple_tv_hype_we_think_not/
winAUTOPWN v2.8 Released For Download – Windows Auto-Hacking Toolkit http://www.northerngrit.com/items/view/3002 I wanted to post this a while back, but the site (and thus the download) was down again – it seems to be a common occurrence. Someone get this guy some proper hosting! winAUTOPWN and bsdAUTOPWN are minimal Interactive Frameworks which act as a frontend for quick systems vulnerability exploitation. It takes inputs like IP [...]

Read the full post at darknet.org.uk

]]>
Tue, 18 Oct 2011 19:27:00 +0200 http://feedproxy.google.com/~r/darknethackers/~3/unUKT63obxs/
Discovering What Facebook Knows About You http://www.northerngrit.com/items/view/2989 Things are getting interesting in Europe:

Max is a 24 year old law student from Vienna with a flair for the interview and plenty of smarts about both technology and legal issues. In Europe there is a requirement that entities with data about individuals make it available to them if they request it. That's how Max ended up with a personalized CD from Facebook that he printed out on a stack of paper more than a thousand pages thick (see image below). Analysing it, he came to the conclusion that Facebook is engineered to break many of the requirements of European data protection. ...

The logical next step was a series of 22 lucid and well-reasoned complaints that he submitted to the Irish Data Protection Commissioner (Facebook states that European users have a relationship with the Irish Facebook subsidiary).

]]>
Tue, 18 Oct 2011 13:34:00 +0200 http://www.schneier.com/blog/archives/2011/10/discovering_wha.html
VMware profits more than double in Q3 http://www.northerngrit.com/items/view/2986 Looks like Microsoft, slurps cash like Microsoft Server virtualization and soon-to-be cloud juggernaut VMware continues to steamroll over its virtualization rivals and bankroll revenue growth and stunning profit growth this year.…

]]>
Tue, 18 Oct 2011 12:19:00 +0200 http://go.theregister.com/feed/www.theregister.co.uk/2011/10/18/vmware_q3_2011_numbers/
Dell ditches EMC after 10 years http://www.northerngrit.com/items/view/2987 Will push own storage kit after ending relationship Dell has officially stopped reselling EMC kit and will instead push its own storage kit. Let battle be joined.…

]]>
Tue, 18 Oct 2011 11:38:00 +0200 http://go.theregister.com/feed/www.theregister.co.uk/2011/10/18/dell_ditches_emc/
Apple and Samsung discuss... CPU production deal http://www.northerngrit.com/items/view/2988 Korean spin suggests A6 producer not picked Samsung and Apple may be suing each other left, right and centre, but that hasn't stopped them talking about chip production partnerships.…

]]>
Tue, 18 Oct 2011 11:27:00 +0200 http://go.theregister.com/feed/www.reghardware.com/2011/10/18/apple_and_samsung_discuss_a6_cpu_production_says_mole/
My Council Services http://www.northerngrit.com/items/view/2990 Hello, I’d like to complain about... Android App of the Week  My Council Services is an interesting little app designed to let people conveniently report issues with public services or the immediate environment to their local council.…

]]>
Tue, 18 Oct 2011 11:00:00 +0200 http://go.theregister.com/feed/www.reghardware.com/2011/10/18/app_of_the_week_android_my_council_services/
Apple iPhone 4S http://www.northerngrit.com/items/view/2991 Enough of an upgrade? Review  Apple’s design language is easily recognisable: clear glass or plastic that’s spray-painted from the inside, a frequent use of aluminium, stainless steel and any colour so long as it’s white or black. But the new iPhone takes this familiarity further, by making the iPhone 4S almost identical to last year’s iPhone 4.…

]]>
Tue, 18 Oct 2011 08:00:00 +0200 http://go.theregister.com/feed/www.reghardware.com/2011/10/18/review_apple_iphone_4s_smartphone/
4chan founder bashes Facebook, Google+ on identity http://www.northerngrit.com/items/view/2992 Facebook = the fast food of identity, Moot moots Web 2.0 Summit  The founder of internet forum 4chan and media-sharing service Canvas has accused Facebook and Google+ of fundamentally misunderstanding how we use identity.…

]]>
Tue, 18 Oct 2011 06:09:00 +0200 http://go.theregister.com/feed/www.theregister.co.uk/2011/10/18/4chan_bashes_facebook_google_identity/
Hacktivists pose growing threat to industrial computing http://www.northerngrit.com/items/view/2993 DHS warns nation about Anonymous Members of the Anonymous hacking collective are increasingly interested in attacking industrial control systems used to automate machinery used by factories, power stations, water treatment plants, and other facilities critical to national security, the Department of Homeland Security warned last month.…

]]>
Tue, 18 Oct 2011 02:03:00 +0200 http://go.theregister.com/feed/www.theregister.co.uk/2011/10/18/anonymous_threatens_scada/
LibreOffice plans ports to iOS, Android, cloud http://www.northerngrit.com/items/view/2994 French government says 'oui' to open source The Document Foundation, which is developing the LibreOffice software suite, has demonstrated the business software working entirely in the browser for cloud applications, and has announced that it will also port it to Android and iOS.…

]]>
Mon, 17 Oct 2011 22:39:00 +0200 http://go.theregister.com/feed/www.theregister.co.uk/2011/10/17/libreoffice_porting_ios_android_cloud/
Steve Jobs memorial brings out tech titans... and Bono http://www.northerngrit.com/items/view/2995 Larry Page forgot to put invite in his Google wallet Tech world supremos Larry Page, Bill Gates, Larry Ellison, Michael Dell and Apple's Tim Cook and Scott Forstall made an appearance at the secret memorial for Steve Jobs held in Stanford University yesterday.…

]]>
Mon, 17 Oct 2011 18:32:00 +0200 http://go.theregister.com/feed/www.theregister.co.uk/2011/10/17/high_security_jobs_memorial/
Apple: 4m iPhone 4S handsets sold, thank you very much http://www.northerngrit.com/items/view/2996 Kerching Apple sold 4m iPhone 4S handsets this past Friday and over the weekend, it claimed today.…

]]>
Mon, 17 Oct 2011 15:30:00 +0200 http://go.theregister.com/feed/www.reghardware.com/2011/10/17/apple_says_4m_iphone_4s_handsets_sold/
iPad baby baffled by paper magazine http://www.northerngrit.com/items/view/2997 Infant QA test result: 'This sucks' Video  Youth is not wasted on the young. But apparently paper is, and so are static and archaic publishing models that don't involve pinching and poking. This baby is so used to an iPad that old-fashioned dead-tree media is just baffling.…

]]>
Mon, 17 Oct 2011 13:31:00 +0200 http://go.theregister.com/feed/www.theregister.co.uk/2011/10/17/baby_magazine_ipad/