Embracing Learning

The recent release of SQL Server 2019 offers new and improved features for data professionals. There's already lots of information out there about the latest bells and whistles; here's a link to start with and if you haven't already then you can download yourself a copy and have a good look around. 

If we look at the added functionality or support for different technologies between the 2016 and 2019 releases then clearly there's a lot of learning required to stay up to date; new introductions like Linux, containers, Kubernetes, R, Python, Apache Spark and lots more provide quite a list of new capabilities for DBA's and developers to put to use within their data platforms so gaining a good understanding of these technologies is becoming more and more vital.

But staying up to date can be difficult, especially in a technical landscape that is rapidly evolving as it is in current times but it's also a time where embracing the learning opportunities out there will undoubtedly be massively beneficial for those willing to invest in their career management.

I say "willing" because ultimately it is a choice, and with all the learning material being made available from vendors and from within the community it should be a very straightforward choice to make. Whilst it's not going to be free; it is going to take some time, money or more commonly both, the benefits far outweigh any personal contribution (or if you prefer to view it, sacrifice) that you'll be making. 

The main point that I want to put across is that right now, more than ever, it's become absolutely essential for anyone working in a data environment to take ownership of their personal development.

Some may be unsure where to start, which isn't surprising considering the rate of change but that's no reason to simply sit back either. Have a think about some of the factors that go into learning roadmaps; job requirements, your personal career aspirations, areas of interest, new technologies and of course, marketability all need to be considered (yes, the latter does matter) and then map to what training material is out there. It's virtually impossible to not find what you're looking for and if you're stuck for ideas - hey, ask away!

An early SQL conference.

So if you are just watching and waiting take the release of SQL Server 2019 and use it as the perfect opportunity to kick start your personal development journey; go through the release material, explore the new features and supported technologies, check out the changes to existing features, read blogs, watch videos, buy or download some books and attend events like conferences and user groups - there's plenty of choice out there to fit any favoured learning style and many require nothing more than an hour or two of your time. Above all, investing in yourself is the most rewarding investment you can ever make.

Reasons why DevOps implementations fail.

Over the last few years we have seen a monumental rise in the number of organisations adopting a DevOps working culture. This trend shows no signs of slowing down whatsoever and whilst many are now realising the benefits of these working practices many are also struggling with the adoption and in some cases it's either stopped completely or not even started.

There's a number of reasons of why this happens and I've seen some common causes, which is actually a good thing because we can recognise where these are occurring or even prevent them before they start to become a problem.

No clear direction.

It's very difficult to drive to a destination if you don't know where you're actually going (trust me on this, I've tried it). This is obviously very reasonable advice however many DevOps projects fail because of a lack of direction. It's actually a common issue with many buzzing trends, particularly in IT where organisations rush into a technology stack or movement just for the sake of doing it. Inevitably this often always leads to failure.

Organisations need to fully understand what a DevOps culture is, its objectives and its close relationships with their existing business processes and frameworks. A common misconception is people often viewing DevOps as a direct replacement for ITIL when in actual fact it's more of a cultural change built on top of ITIL principles. By fully understanding DevOps the benefits of adoption become much more viable and ultimately the path to it's success becomes much simpler.     

Adopting a silo approach to DevOps.

I often see individual teams being very successful in implementing components of DevOps practices only for other teams being behind in terms of adoption and/or understanding. The classic case is the Developer and DBA; the developer is pushing for much more frequent releases (a realised benefit of DevOps) but then the DBA, who perhaps isn't on board, is then trying their best to slow down all of these changes to their production databases. In the words of Bon Jovi, "we're half way there".

This lack of cohesion or a shared direction can result in a significant bottleneck and the DevOps practices start to creak a little. Then other unintended side effects start to creep in, such as blame and finger pointing (some of the things that a healthy DevOps culture seeks to eliminate) and then it can all start to fall apart. 

DevOps for databases is one particular area that is so heavily reliant on lots of people in different roles working together in a collaborative manner. An organisation must identify this and start to encourage teams to engage and build with each other in the early phases of a DevOps implementation, but organisations also have to be very careful in how they seek to develop this collaborative culture...

Forced collaboration.

I believe collaboration underpins the entire DevOps methodology so it makes perfect sense for organisations to work towards developing much closer working relationships between teams however organisations can also over-formalise things, even making the activity seem very process-driven which often leads to much less buy in from individuals, even entire teams.

This causes obvious problems, not least the silo approach mentioned in the previous point, so organisations have to find the balance on being almost relaxed in how they let relationships build and at the same time provide a certain degree of steer. This isn't as easy as it sounds and it is certainly reliant on strong leadership. In my experience successful implementations have been led by those that enable positive change rather than those who try to dictate it.

Rushing into new tools.

New tools are awesome, fact and in a DevOps ecosystem there are so many to pick and choose from that each bring new ways of doing things and [hopefully] improving productivity. The advantages are great, without a doubt but often tools can be implemented way too early without a focus on the underlying processes. This can significantly reduce the effectiveness of what a particular toolset/platform is trying to achieve; a release management tool for example won't improve a change/release process if the process is fundamentally flawed.

The most successful DevOps implementations focus on people and process first, leveraging the strengths and principles of existing frameworks and building strong collaborative working practices. New tools are going to be an important factor of a system designed with DevOps principles in mind but they need to be leveraged correctly and for the right reasons.


These are some of the common pitfalls that I've seen occur during DevOps implementations, many of which are deeply rooted in the working culture, not the technologies. There is undoubtedly so much to gain by adopting the principles and often it requires organisations to step back and observe their working practices first and spend time exploring the benefits of working with DevOps.

In my next post I'll cover how to address some of these issues and offer some insights into the benefits of building collaborative relationships between data professionals.

The DBA and the Data Platform

I've been consulting now for well over a year, it's been a fantastic experience so far; I've had the pleasure of meeting and working with some truly incredible people and it's also been amazing to be helping organisations get the most from their SQL Server platforms.

The last 12 months have been extremely interesting considering the current technical shift towards a seemingly endless list of new platforms and capabilities. For many organisations this shift has fundamentally changed how they do things (or going to do things) and many are going through their own journey of digital transformation understanding these new technologies and how to leverage them to improve their existing processes.

Right at the centre is data, nowadays considered to be a companies greatest asset, so how we use data or to be more exact how we extract more value from it is a frequent aim of these transformation projects. As a result the data professional roles within an organisation have changed too and many teams and individuals are having to broaden their technical skill set to design, deliver and support these new capabilities.

There is no truer example of this than the role of DBA but yet the question still remains on which direction(s) they should be focusing their future on? By that I mean development; which skills should they be learning, what certification, what products, platforms etc? It's easy to see why it's confusing; there are lots and lots of new technologies available to us but that also means there is no consistent path for administrators to set out on. 

It was easier in the not so distant past; the DBA role tended to come in two distinct flavours; development or operational, but now the line between the two has became far less apparent and in many cases non-existent. The DBA role has without question become much more rounded containing core elements from both "sides" but it goes beyond that. Modern approaches to database administration have reduced the footprint of the operational side of the role so a more DevOps orientated DBA is an essential part of the roadmap, for organisations and individuals alike.

This shared roadmap is actually an integral component of an evolving data platform. I've worked with many different organisations at different stages of development but a key component in each has always been a collaborative working practice between data professionals and I'd say the success of a platform depends on it. Over the past year I've worked alongside many different teams; architects, DBA's, developers, release analysts, change and testing specialists, when working together they all have a great influence of the effectiveness of a data solution. 

Whilst it's fair to say organisations should be encouraging this type of working practice it is very much dependent on individuals to pursue and implement it. For DBA's this means engaging with other teams much more frequently, understanding their skills and processes but also adding their own expertise and skill set to produce a much more co-operative function. 

It might not fully answer this common question of what next for DBA's, there's a lot of factors involved, not least your personal career aspirations but it does show how when looking for your next phase of self development the paths that you need to take could be very close to your current role. 

Your skills as a DBA will always be of paramount importance towards availability, security and performance, but by widening your technical scope by working much more closely with those around you not only improves you as an data professional but also the overall effectiveness your data platform. The starting point is to seek out these opportunities; start building relationships, start knowledge sharing and developing new ideas and different ways of working. 

In many ways the data platform is a reflection of the people tasked with shaping it. If we choose not to broaden ourselves, learn new ideas then at some point we come to a standstill, especially in this rapidly evolving technical landscape. If that happens a data platform stops growing, not in terms of data quantity but most certainly in terms of both capability and perhaps most critical of all, value. 

This puts a big emphasis on self-development but we've always had that, it's nothing new. The key is to seek out those opportunities, start close and work with teams that you will probably already have a working relationship with. As these relationships grow so does the collaborative skill set and as a result business processes improve and technical solutions get faster and more productive.

This also cuts down reactive measures, less time fire fighting means more time delivering enhancements and of course, more learning. Any learning process shouldn't have a cut off point, the more we broaden our horizons the more the effective we become and in turn, our products, our services and our platforms keep on evolving.

This is the true nature of a technical platform and we as data professionals play such an important part not just in the support or development of them but towards shaping their growth, effectiveness and their value. Perhaps the key to all of this is realising the value of our own development alongside it.   

Installing Python and using the new command line T-SQL script generator

Earlier today two new command line tools were announced for SQL Server, one an experimental Linux tools DBFS which enables access to live DMVs without using a UI like SSMS and secondly a tool that enables script generation of objects within SQL rather like the Generate SQL Scripts option in SSMS.

In this post I'm going to run through the installation of the script generator tool and provide a very quick demo. The reason I'm going through this is because in order to install the tool we need to use something called PIP. PIP is a package management system that enables us install and use packages written in Python. Yeah, Python again! 

So if you haven't already installed Python, because it isn't available by default in Windows you'll need to get it from here:

https://www.python.org/downloads/

This will take you to the download screen and here you get two version options; 2.7.13 or 3.6.1, I went for the latter and there is a link that explains the differences between version 2 and version 3 of Python: https://wiki.python.org/moin/Python2orPython3

And this is the installation screen, before hitting the install option make sure the Add Python 3.6 to PATH is selected otherwise things won't be working as expected after install.

Nothing untoward in the actual installation and we get a successful message in a couple of minutes.

So now that Python is installed we need to install the mssql-scripter package and that is very straightforward and rather familiar to those using PowerShell I guess. We use the command: pip install mssql-scripter from a Windows command prompt ran as administrator.

The installation package goes and does its thing!

Again, after a couple of minutes we get a couple of done messages and returned to the command prompt, installation all good.

Once complete we can pass in a -h parameter to the mssql-scripter command and see a wide array of options that are available to us but in this demo I'll keep it simple and you can refer to the original link to see the different ways of using the tools.

To use the command I've typed out the following which will create a script (adventureworks.sql) in my working directory containing the DDL and DML commands for the objects in the AdventureWorks2014 database:

mssql-scripter -S localhost -d AdventureWorks2014  > ./adventureworks2014.sql

Worth noting that although mssql-scripter is brand new I am using this on a SQL Server 2014 instance. 

The script ran in just a few seconds which is very impressive and it produced the file as expected and whilst I'll not put the entire script content in this post here is a screenshot of the script opened inside SSMS.

Now I did work through some of the examples on the release article however I did run into unrecognized argument errors when using -schema-and-data which is shame as I did want to test that functionality. I'll await a comment back on that one and then perhaps run through some more scenarios of using the command.

Edit: I did parse the created script: 

Command(s) completed successfully.

Torn Pages and using DBCC PAGE

Today's post is about torn pages and using the undocumented DBCC PAGE command within SQL. Occasionally you might find you need to run this command and in this example it's because I have received an error message that looks a bit like this:

SQL Server detected a logical consistency-based I/O error: torn page (expected signature: 0xaaaaaaaa; actual signature: 0xa9aaaaaa). It occurred during a read of page (1:25756919) in database ID 9 at offset 0x000031209ee000 in file 'F:\DATA\ScratchDatabase.mdf'

This is an error that has been picked up on one of my test systems and indicates that SQL Server has detected a torn page, that is a page that has been incorrectly written by SQL Server and possibly indicates a problem in the IO subsystem.

The problem here is that whilst we know the database and the page where the error has occurred we don't know the specific table the page belongs and importantly what type of page is in error. The reason why the page type is important is because this will drastically impact our recovery process but the first thing we will do is check a system table to see if any other page errors have been reported:

SELECT * FROM msdb..suspect_pages

database_id	file_id	page_id	event_type	error_count	last_update_date
9	1	25756919	3	1	2017-01-29 08:24:05.760

Here we can see that only one row has been returned, the event_type refers to a torn page, here's the full type descriptions and a link for more information:

1 = An 823 error that causes a suspect page (such as a disk error) or an 824 error other than a bad checksum or a torn page (such as a bad page ID).

2 = Bad checksum.

3 = Torn page.

4 = Restored (page was restored after it was marked bad).

5 = Repaired (DBCC repaired the page).

7 = Deallocated by DBCC.

To identify the table in error we will use DBCC PAGE. This command takes a database id (or name), file id and page id and will return the actual page information back to the results window (we need to enable trace flag 3604 to do this though); for my example I would use the following command:

DBCC PAGE (9, 1, 25756919, 0)

This returns a lot of information to the results window (I can use the option WITH TABLERESULTS for a grid view) so I have removed a fair chunk of it and highlighted the two parts that I really need.

PAGE: (1:25756919)

 

PAGE HEADER:

Page @0x0000000728010000

m_pageId = (1:25756919)             m_headerVersion = 1                 m_type = 2

m_typeFlagBits = 0x0                m_level = 0                         m_flagBits = 0x104

m_objId (AllocUnitId.idObj) = 266296m_indexId (AllocUnitId.idInd) = 256

Metadata: AllocUnitId = 72057611489902592                               

Metadata: PartitionId = 72057607060652032                                Metadata: IndexId = 20

Metadata: ObjectId = 567673070      m_prevPage = (1:25756918)           m_nextPage = (1:38369560)

The first is Metadata: ObjectId = 567673070 which refers to the table that this page belongs to. The second is Metadata: IndexId = 20 which tells me as it isn't 0 (heap) or 1 (clustered index) that the page error is within a non-clustered index and I can use the following code to identify the actual index where the problem lies.

SELECT * FROM sys.indexes WHERE index_id = 20 AND object_id = 567673070

This returns the suspect index and to resolve we can very simply drop and recreate the index effectively reallocating the pages, suspect and all. This is potentially much quicker than performing a full restore and I have lost count of the times I have seen people resort to a full restore without checking the actual type of page corruption that they're attempting to resolve.


Another option here is that because we know the effected table we can run a DBCC CHECKTABLE command (rather than a DBCC CHECKDB on the entire database), passing in the full table name as a parameter; to get the name of the table we use our ObjectId again that was returned from DBCC PAGE with the following:

SELECT OBJECT_NAME (567673070)

This would return the table name that we can use to check the structure to see if our page issue has been resolved:

DBCC CHECKTABLE ('dbo.ETL_TestLoad) WITH NO_INFOMSGS

DBCC results for 'ETL_TestLoad'.
There are 341465 rows in 93513 pages for object "ETL_TestLoad".
DBCC execution completed. If DBCC printed error messages, contact your system administrator.

Using DBCC CHECKTABLE in this scenario may have proven that our torn page has been resolved but you should always, without question, perform a full database consistency check to look for further errors. In the event of IO issues this could easily return more and more page problems, maybe irrecoverable data page ones at that. Any page read/write errors must result in both a full consistency check and a check with the storage people to see if they can see any underlying issues.


For some further reading check out my post: How to consistency check large databases

22/01/2017 - What I've been reading.

A couple of posts I'd like to mention. The first is Killing Databases in SQL Server on Linux from Andrew Pruski (twitter) - very interesting read about the behaviour of Linux and a particular quirk that was discovered. The comments make an interesting read too but I'm not going to go into those. What I will say is that I've had a good read of other articles on Andrews blog, I really like the style of it with the recommended reads and especially the coffee bits. 

The next post is from Edwin Sarmiento (twitter) entitled Confessions of a Junior SQL Server DBA. Edwin's posts are always thought provoking and this one is really no exception. It made me think back to when I was starting out with SQL Server and all the people that have helped me along the way, there's a mention of Aaron Bertrand that just typifies the SQL Server professional.

Have a read and like most SQL blogs have a good read through the blogs because there's an awful lot to discover in them!

Question on DBCC DROPCLEANBUFFERS

This is a hastily written post in progress but I'm putting it out there anyway. I've just asked via #sqlhelp on Twitter why that I after I run a DBCC DROPCLEANBUFFERS command I can then query the memory clerk and buffer descriptor DMV's and still see allocations to my database.

Here's what I've been doing:

First, run a checkpoint command to write any dirty pages prior to running DBCC DROPCLEANBUFFERS to remove all the buffers from the buffer pool.

CHECKPOINT

DBCC DROPCLEANBUFFERSAfter that we'll query the memory clerks DMV to see how memory is being allocated in my instance:


SELECT TOP(10) [type] AS [ClerkType],
SUM(pages_kb) / 1024 AS [ClerkMb]
FROM sys.dm_os_memory_clerks WITH (NOLOCK)
GROUP BY [type]
ORDER BY SUM(pages_kb) DESC OPTION (RECOMPILE);

What I can see here is that 2Mb is allocated to the Buffer Pool.

I can drill into this in a bit more detail by querying the buffer descriptors DMV where we see the page allocation detail per database.

SELECT DB_NAME(database_id) AS DatabaseName,

page_type, COUNT(page_id) * 8 As SizeKb

FROM sys.dm_os_buffer_descriptors

GROUP BY page_type, DB_NAME(database_id)

In this case the AdventureWorks2014 database has 360kb of data pages and 168kb of index pages; out of interest the NULL DatabaseName rows refer to the ResourceDb.

So my question is, what is persisted in memory after I have dropped the buffer pool?!

UPDATE:

Spoke to a couple of people via #sqlhelp on twitter. We're none the wiser really and I will come back to this and use DBCC Page to look in a bit more detail but for now its a safe assumption that the pages in memory are there and related to SQL Server "doing things" behind the scenes!