Solucionando problemas do SQL Server 2005 Integration Services Pedro Antonio Galvão Junior Latex São Roque – Analista de Sistemas/DBA MVP – Windows Server System – SQL Server pedrogalvaojunior@gmail.com

Agenda O que é Integration Services? Arquitetura. Perguntas Freqüentes. Benefícios do SSIS. Ferramentas SSIS. Demo. Overview of SSIS. Entradas de pacotes filhos em pacotes pais. Usando Breakpoints. Log Data para Troubleshooting. Demo Conclusão.

O que é Integration Services? Plataforma para construção das novas soluções de integração de dados Sucessor do DTS – Data Transformation Services Parte da plataforma de Business Intelligence da Microsoft

Arquitetura Separação entre o Work flow e o Data Flow Views separadas Data Transformation Runtime (DTR) Control Flow = Runtime = Task Data Transformation Pipeline (DTP) Data Flow = Pilepine = Transforms Views separadas

Data Destination Adapters Arquitetura Standard transforms Custom transforms Data Destination Adapters Data Source Adapters XML Package Sequences Loops Tasks Event Handlers DTS Designer Wizards Command Line

Antes do Integration Services Notificação Data mining Dados semi estruturados Legado: Arquivos Bancos de Dados Decodi- ficação Staging Mineração ETL Limpeza & ETL Staging ETL Warehouse Relatórios Acesso Móvel Integração e warehousing requerem operações e estágios separados Preparação de dados requer diferentes ferramentas Grandes volumes de dados tornam o processo mais demorado

Com o Integration Services Notificação Warehouse Relatórios Acesso Móvel Integration Services Mineração Outras Fontes Padrão Limpeza Fusão Dados semi estruturados Legado: Arquivos Banco de Dados Integração e warehousing faz parte da mesma operação Extração, transformação e carga no mesmo processo (aditáveis) Notificação pode compor uma tarefa dentro do package Escalável

Como o SSIS Funciona Podem ser escolhidas vários Data Sources Possibilidade de transformação utilizando vários formatos diferentes Dados são roteados obedecendo regras de negócio ou condições de erro Implementa concorrência A carga pode ser feita em várias destinos paralelamente

Perguntas Freqüentes Por que a execução dos pacotes é tão lenta? Por que eles utilizam tantos recursos? Por que eles não utilizam determinados recursos Por que eles não utilizam mais recursos Devo usar o banco de dados ou o SSIS? Como eu carrego dados mais rápido no SSIS? Tenho mais parâmetros para configurar? O SSIS é mais rápido que o DTS? Quais recursos estão disponíveis?

Benefícios do SSIS Performance Facilidade Agilidade Data Flow manipula grandes volumes de dados de forma mais eficiente Suporte a operações mais complexas Facilidade Vários adaptadores disponíveis reduzem o “hand coding” Extensibilidade habilita o desenvolvimento dos seus próprios adaptadores que funcionaram como se fossem nativos Ambiente visual mais produtivo com suporte a debugging Agilidade A limpeza de dados pode ser feita durante a carga Facilidade no tratamento de regras de negócio dentro do workflow

Ferramentas do SSIS packages Installer File set deploy BI Studio Deployment BI Studio SSIS packages packages Business Intelligence studio is the graphical SSIS package designer (as well the designer for RS and AS). The designer allows access to the import/export wizard as well as building the deployment file sets Dtexec.exe is the main \ production time execution utility, an example is….. dtexec.exe /FILE C:\_WscopeSample\Worldscope.dtsx /REPORTING N Dtexecui.exe allows for adhoc execution as well as aiding the ‘construction’ of the dtexec.exe command line. Dtexecui is exposed in Mgt Studio, by double clicking a pacakge file (.dtsx) and you can ‘run’ the file “dtexecui.exe” Dtutil.exe as the name implies is a utility, for various package operations (not execution). For example It will allow me to copy packages, sign the, and change the encryption type\level. An example is….. dtutil /FILE C:\_DirectoryList\DirectoryList\DirectoryList\DirectoryList.dtsx /DestServer craigg02 /COPY SQL;DirectoryList This sample copies a package from the file system to SQL with name ‘DirectoryList’ See BOL topic regarding Command Prompt Utilities when filtering for just SSIS topics Import Export Wizard Dtutil.exe execução View running packages import\export Dtexec.exe SSIS Service Mgt Studio Dtexecui.exe

SQL Server Business Intelligence Integração Análise Report Aquisição de dados e integração. Transformação de dados. Enriquecimento dos dados, com Lógica de negócios Data mining Apresentação de dados e distribuição. Acesso de dados para a massa

demo SSIS Overview Tour pelo Business Intelligence Studio <SLIDETITLE>Demonstration: {Demonstration Title}</SLIDETITLE> <KEYWORDS></KEYWORDS> <KEYMESSAGE>.</KEYMESSAGE> <SLIDEBUILDS>0</SLIDEBUILDS> <SLIDESCRIPT> </SLIDESCRIPT> <SLIDETRANSITION></SLIDETRANSITION> <ADDITIONALINFORMATION> <ITEM></ITEM> </ADDITIONALINFORMATION> SSIS Overview Tour pelo Business Intelligence Studio Connection Manager

Segurança no SSIS Composta de algumas camadas para suportar cenários SQL e arquivos externos. Os Packages podem ser criptografados ou assinados por certificado digital. Packages podem ser armazenados em bancos SQL Server e protegidos com SQL roles. SSIS is somewhat unique in that it needs to support ‘secure’ and flexible options for packages and execution when stored in SQL or on just the File system. As there are several layers to security model it is good to be aware of them before deploying packages. For example the default protection level EncryptSensitiveWithLocalUserKey, the key being a combination of the current user and current machine. Therefore if I move the package from 1 machine to another I will get key errors opening the package, even if I am the logged into the domain as the same user. The key part of the deployment plan or ‘hand off’ plan from dev to test to production should be an agreement on what protection level packages are developed with and what IT/Operations teams should change them to for production.  For example they could use a password protection level and when things are handed off to the production team they change the password.  Or say the production team will take the packages and ‘upload’ to the production SQL box and at that time change the protection level to server storage, control limit access via the SQL role and now dev\test cannot touch the package.  There is also the ‘double protection’ option where the package can be stored in SQL (so access limited by SQL roles) AND they can still have it encrypted with a password and key if they want.  The double option actually allows for the scenario where you can even limit the SQL DBA from getting to the package as they would not know the password or proper user\machine key. All sounds great, but there is some overhead when dealing with a lot of packages. So again an agreed plan helps. For example it could be as simple that everyone develops with the default local key option so its easy for them (e.g. no passwords or uploading\downloading from SQL store) and before they hand off to test\operations they encrypt the package with a password which the receiving team knows and now the local key is not an issue. Remember you can batch edit with DTUTIL.EXE 2 examples **can add the /q switch to avoid extra prompts…and the double %% is used in batch files while single % at the command line itself. 1) first one is an example what devs can do before handing off to operations…so developer left was using the package with the default encrypt with user key…now they are ready to hand off and encrypt with a password the operations team will know ‘topertions’, this assumes they have many packages they want to hand off otherwise they can just do it inside designer rather than a batch file. 2) the 2nd one is what the ops person would do, loop again and change all to a password the dev team does not know “productionpassword”.  This of course is just an example, they could loop and encrypt everything with the current local key..say while logged in as the account they is going to execute jobs. rem on developers machine where they do not need to decrypt as default was local key, but encrypting to password key ops will use. rem Level 3: Package is encrypted using the required password (tooperations) for %%f in (*.dtsx) do dtutil.exe /File %%f /encrypt file;%%f;3;tooperations rem now operations from a machine of their choice decrypts password 'tooperations' to password 'productionpassword' (option 3 is encrypt whole package. for %%f in (*.dtsx) do dtutil.exe /File %%f /Decrypt tooperaations /encrypt file;%%f;3;productionpassword

Segurança no SSIS Package Protection Level Property Roles de leitura e escrita nos pacotes armazenados no banco MSDB SQL DB Roles A Permissões NTFS Criptografa ou limpa propriedades sensíveis B Assinatura do pacote Controle de acesso a pastas ou nos arquivos dos pacotes In SQL Books Online (BOL) review the topics ‘Setting the Protection Level of Packages ‘ ‘Security Considerations for Integration Services ‘ (A) Um pacote pode ser criptografado via Package Protection level ou armazenadp dentro do MSDB, onde o acesso ao pacote é limitado pelos SQL Database roles (B) Um pacote pode ser criptografado pelo Package Protection level ou o arquivo do pacote pode ser controlado por permissões em pastas e arquivos (NTFS).

SSIS Windows Service Monitora e gerencia pacotes rodando ou armazenados, através do SQL Management Studio O Serviço é instalado quando instalamos o SSIS O Serviço não é necessário para design ou execução dos pacotes Os pacotes armazenados (Stored Packages) é baseado em um arquivo XML. Podemos customizar conteúdos de arquivos, nomes e localidade Tem eventos Windows para o serviço, ou seja, se se o serviço inicia, pára, falha ao iniciar, iniciou um pacote , um pacote parou …) In Books Online (BOL) review the topics Managing Integration Services Service Troubleshooting Integration Services Service The SSIS service is the component in SQL 2005 that monitors and manages running as well as stored packages, via SQL Management Studio The service is installed when you install SSIS. It is not required to design or execute Packages. Stored Packages tree is based on XML configuration file that the service reads on start. You can customize the file contents, name, and location A registry setting tells the service where and what configuration file to use Windows events written for service (service start, service failed to start, package started, package stopped…) Since the config file is based on a registry location, you could point all servers to the same file on a central share. Service posts windows events to the application log…for example an event is posted on service start noting the version. There are also hard coded events (info) posted for package start and package completed. Intent is for Operations safety valve.

Posso criar meu próprio client Quero criar uma console administrativa em asp.NET. Como posso listar os pacotes? Podemos fazer isso através do service interface no Application Object. static void Main(string[] args) { Application app = new Application(); PackageInfos pInfos = app.GetDtsServerPackageInfos(@"File System", "yourserver"); foreach (PackageInfo pinfo in pInfos) Console.WriteLine(pinfo.Folder + " " + pinfo.Description); } See the SSIS forum on MSDN for more great information… http://forums.microsoft.com/msdn/ShowPost.aspx?PostID=15704 Yes, you can do this through the service interface on the Application Object. Something like this give you all the packages on the SQL Server instance. : Dim app As Application = New Application() Dim pInfos As PackageInfos = app.GetPackageInfos("\\","yourserver",Nothing,Nothing) For Each pInfo As PackageInfo In pInfos Console.WriteLine("Name: {0}", pInfo.Name) Next For the packages on the IS Server, do something like this: static void Main(string[] args) { Application app = new Application(); PackageInfos pInfos = app.GetDtsServerPackageInfos(@"File System", "yourserver"); foreach (PackageInfo pinfo in pInfos) Console.WriteLine(pinfo.Folder + " " + pinfo.Description); }

demo SSIS Overview Mostrar Log Provider Mostrar Log Events <SLIDETITLE>Demonstration: {Demonstration Title}</SLIDETITLE> <KEYWORDS></KEYWORDS> <KEYMESSAGE>.</KEYMESSAGE> <SLIDEBUILDS>0</SLIDEBUILDS> <SLIDESCRIPT> </SLIDESCRIPT> <SLIDETRANSITION></SLIDETRANSITION> <ADDITIONALINFORMATION> <ITEM></ITEM> </ADDITIONALINFORMATION> SSIS Overview Mostrar Log Provider Mostrar Log Events Progress Pane/Output 18

Log Data para Troubleshooting Logging e Error Flow são principais recursos para troubleshooting Pode ser salvo e carregado Package IDs Necessitam ser únicos. Se precisar, poderá re-gerar o ID via BIDS ou dtutil.exe Exemplo: O comando abaixo irá regenerar o ID do pacote para todos os pacotes numa pasta. for %%f in (C:\_work\SSISPackages\_quick\Notepad\*.dtsx) do dtutil.exe /i /File %%f Log entries are a blend of status and result messages For logging you can select what ‘details’ per control flow object within each package (e.g. OnError, OnWarning, OnPreExecute) Can define multiple log providers (SQL, text file, Windows Event..) per package One suggested configuration is to have all of your packages write to a central SQL server using the SQL log provider. This will allow operations staff to monitor log entries from all packages in a central location. Error flows could also be centralized to a SQL DB, and its handy to add an audit transform. You can also create your own custom log provider. Perhaps you have a custom monitoring program that consumes files of specific format. You could create your own log provider which saves the log entries in the desire format. NOTE: Packages have an ‘ID’ property which is a GUID and uniquely identifies the package, you can see this as the sourceId Package IDs and the need for keeping the ID unique…and the ability to re-generate the id via BIDS or dtutil.exe So the intent is it will regenerate package IDs for all packages in a folder…be it 2 or 200. for %%f in (C:\_work\SSISPackages\_quick\Notepad\*.dtsx) do dtutil.exe /i /File %%f Tip: Logging is great but turning everything on for every package can affect package perf, it’s a lot of data Tip: an agreed naming convention for object names can be helpful when you start to get a lot and want to sort out in log data. The ‘name’ property is seen as the ‘source’ field in SQL logging data. Tip: the Package description property is optional and empty by default. It’s a good practice to provide an description as it will appear in Mgt Studio when you view the details of a running package. Tip: Not easy but its possible to parse out the new numbers in the onpipepline rows sent log entry to get status of rows processed. The idea being you could have a report periodically refreshing showing the latest numbers.

Entradas de pacotes filhos em pacotes pais Event Source SourceID ExecutionID User:OnPreExecute Child1 12460949-3505-4 57CDDEAE-681B-4 User:OnPostExecute Execute Package Task_child1 B2488939-1412-4 OnPostExecute Execute Package Task_parent BD73EC3F-ED36-4 OnPreExecute Parent 3EC3B689-C151-4 PackageEnd PackageStart Send Mail Task_child1 0B513B33-72A2-4 Send Mail Task_parent 4D6E4F65-7A57-4 The child package will still log the same entries under its own executionID, however they are also duplicated to the parents log entry data with the parents executionID. IF this was not done it would be impossible to know what set of entries from the child were based on what entries or execution of the parent.

Usando Breakpoints Configurar breakpoints na caixa de diálogo Set Breakpoint Especificar as condições de break Desabilitar e apagar os breakpoints

Conclusão Devemos utilizar o SSIS! Ferramentas do SSIS Utilizando o SQL Server Bussinnes Inteligence Segurança dos pacotes Usando Logs e breakpoints para tratar os erros

Seu potencial. Nossa inspiração.MR © 2006 Microsoft Corporation. Todos os direitos reservados. O propósito desta apresentação é apenas informativa. Microsoft não faz nenhuma garantia expressa ou implícita nesta apresentação. Seu potencial. Nossa inspiração.MR