Tools | Data Tools and Software

Tools, Data Engineering  · 

“To the man who only has a hammer, everything he encounters begins to look like a nail.” - Abraham Maslow

Today’s Data Engineer requires a variety of tools for their day to day tasks, which vary from working with source control to containers. The objective of this post is to share with you the various tools that I use.

Integrated Development Environment (IDE)

Code IDE’s

  1. Visual Studio 2019 community (latest) here
  2. Visual Studio Code here
  3. SQL Server Data Tools (latest) here

Database IDE’s

  1. SQL Server Management Studio (SSMS) (latest) here
  2. Azure Data Studio (latest) here
  3. MySQL WorkBench (latest) here
  4. pgAdmin-4 (for Postgresql) here
  5. Oracle Sql Developer here
  6. MongoDB Compass (appropriate version) here
  7. PAID License Multi-platform | DataGrip here

RealTime Data analytics’s

  1. Azure Service Bus Explorer ServiceBusExplorer-*.*.*zip here

Data Science Tools

  1. R-Studio here
  2. Anaconda here - This is a collection of DS tools

SDK’s & Plugins

  1. .Net core here
  2. Python 2.7.x here
  3. Python 3.x (latest) here
  4. NodeJS (latest) here

Infrastructure

  1. Docker here
  2. minikube here
  3. Azure Storage Emulator here
  4. Azure Cosmos DB Emulator here

DevOps

  1. AzurePowershell RM here
  2. Az CLI module here
  3. Chocolatey here
  4. flywaydb Community Edition here

Utilities

  1. Notepad++ (latest) here
  2. DiffTool Winmerge here
  3. cmder here
  4. git here

Powershell Modules

  1. dbatools here
  2. posh-git here
  3. MicrosoftPowerBIMgmt here
  4. ~AzureRM Powershell Module (see above under DevOps)~
  5. Azure AZ Powershell Module here

Useful Websites

  1. https://statisticsparser.com/ - SQL Server SET STATISTICS * parser
  2. https://www.brentozar.com/pastetheplan/ - Share SQL Server Execution Plans
  3. https://jsoneditoronline.org/ - Edit JSON Files easily without worrying about escapes.
  4. http://www.kloth.net/services/nslookup.php - DNS Lookup Site
  5. https://stackedit.io/ - Online Markdown Editor.
  6. https://colab.research.google.com/ - Google Colab (Jupyter notebook like Google Docs)

Slack Communities

  1. SQL Server Slack Community https://dbatools.io/slack
  2. Powershell Slack Community http://slack.poshcode.org/
  3. Octopus Deploy Slack Community https://octopus.com/slack

Personal Notes

Azure Data Studio

Currently, I have a love-hate relationship with Azure Data Studio (Previously SqlOps Studio). While it has a long way to go, the features such as inbuilt graphs, exports, temperamental auto-complete and plugins - do differentiate it slightly, however, I do have to push myself to use it and when I do, I usually end up back in SSMS to do some serious work.

JetBrains DataGrip

Whilst it is an excellent tool with good support for connecting to multiple systems, I have not been able to justify the need for it - this is probably attributed to the fact that I work mostly on SQL Server and Microsoft Tech.

Visual Studio Code

This is one of the IDE’s I turn to on a regular basis, whether it is to blog (Jekyll), script (Powershell, Shell, Batch), develop (Python, C#), document (Markdown), Infrastructure (Azure ARM, Docker) - VSCode has a plugin for it. It even has extensions for Data but I always go to SSMS - ‘old habits die hard’

My Plugin List:

Requires code to be part of your PATH

code --install-extension Compulim.vscode-azure-storage-utility
code --install-extension DavidAnson.vscode-markdownlint
code --install-extension eamodio.gitlens
code --install-extension eriklynd.json-tools
code --install-extension fcrespo82.markdown-table-formatter
code --install-extension formulahendry.azure-storage-explorer
code --install-extension msazurermtools.azurerm-vscode-tools
code --install-extension ms-azuretools.vscode-azureappservice
code --install-extension ms-azuretools.vscode-azurefunctions
code --install-extension ms-azuretools.vscode-azurestorage
code --install-extension ms-azuretools.vscode-azureterraform
code --install-extension ms-azuretools.vscode-cosmosdb
code --install-extension mshdinsight.azure-hdinsight
code --install-extension ms-kubernetes-tools.vscode-kubernetes-tools
code --install-extension ms-mssql.mssql
code --install-extension ms-python.python
code --install-extension ms-toolsai.vscode-ai
code --install-extension ms-vscode.azure-account
code --install-extension ms-vscode.azurecli
code --install-extension ms-vscode.csharp
code --install-extension ms-vscode.PowerShell
code --install-extension ms-vscode.Theme-MarkdownKit
code --install-extension ms-vscode.vscode-azureextensionpack
code --install-extension ms-vsts.team
code --install-extension PeterJausovec.vscode-docker
code --install-extension redhat.vscode-yaml
code --install-extension Summer.azure-event-hub-explorer
code --install-extension usqlextpublisher.usql-vscode-ext
code --install-extension VisualStudioOnlineApplicationInsights.application-insights
code --install-extension vsciot-vscode.azure-iot-edge
code --install-extension vsciot-vscode.azure-iot-toolkit
code --install-extension wmontalvo.vsc-jsonsnippets
code --install-extension yzane.markdown-pdf