Tuesday, December 3, 2013

SSIS:WEB SERVICE

Source: SSIS Script Component Transformation Video SSIS 2012

1.Control Flow:Full Business Logic.

2.Data  Flow:

i.Real "Source" to "destination" data flow happens.

ii."Transformations" before inserting into Destination.

OLEDB Source

Script Component

Derived Column (enable "Data Viewer")

3.In this lesson "Script Component Transformation" example is shown.

 i.In "Script Component Transformation" , one can choose as "Source","Destination" or "Transformation".

 ii.Choose "Transformation".

Script Language:Visual Basic 2010
                                      or  Microsoft Visual C#  2010

Mapping
Input
Output
Create any "output column"

Choose button "Edit Script" -

        Navigated to "Microsoft Visual Studio".

Public strvar1 As String
Public strvar2   As Decimal


Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)

'Add Code here

If  condition Then

.....
Else
....

End If
.....
End Sub

End Class
------------------------------------------------------------------------------------------------------------

Source:Consuming a web service in SSIS 2

Data Flow:

 Script Component( prompts: choose "source")

  Right Click( on above component)

  1.(Custom Properties:script Language: dropdown(Microsoft Visual Basic 2005,Microsoft Visual C# 2005)- click on "Edit Script" button,it lauches the Visual Studio.

i.Define "Output Column":what the webservice returns (name,city etc)


Note:In 2008 version the "Script Language" is new)(Lauching Visual Studio is also new).


 2.( takes to Visual Studio: Reference:"Add Web Reference" (prior to this :copy the url from web service) - click(Add Reference)

  i.in visual studio the main.cls click the method(CreateOutputRows)
    To use it ass

     using (click on object explorer tab, copy the webservicename)

     using system.xml;

     ...
     Note:Inside method(CreateOutputRows)

           Note:Define the object of that WebService like

           train mytrain= new train();

  string myresponse= mytrain.gettraindestination(xyzparam);

Note:In 2008 version "Add Web Reference" also new.
Note:It adds as a 'Proxy Class',enable it communicate with that Web Service.

Run to build it.

Back to BIDS.

3.Add component "Row Count" and chk. Variable and a "Data Viewer", see the data.

------------------------------------------------------------------------------------------------------------

Source:SSIS 2008 Series:Using Webservices in SSIS Packages

1.Start BIDS:just create a SSIS project.
2.In Visual Studio: create a Project ->"ASP.NET Web Service Application" project.
  i.change say the message

...
[WebMethod]

public string Hello World( )
{
return "Hello World";
}

a. Run the package , it will return the XML

  ii.The Web Service is created

      Service1
HelloWorld
Invoke

Click on invoke , will show the string.

Change the url adding at the end "WSDL" : http://localhost:55345/Service1.asmx?WSDL

give the whole - XML

        Click on the "Service Description"  , save it as a link ( in a folder WebSrvr.asmx)

Note:Generally before using the Web Service , it's hosted in IIS, but for this example (Visual Studio running & Dev. running )



3.In BIDS create new connection :(HTTP)

  Server URL:WSDL path( the step 3 link which is saved , rename it to WSDL)

  ( you can use proxy but for now the above)

4.Now in "Control Flow" put task "Web Service Task" , specify the connection (HTTP one), specify the WSDL file,set "OVerwriteWSDLFile:True"

Input:

Service:Service1

Output:

OutputType: File Connection ( Can be "variable" also.)

Create File

Note:If you put in variable using "Script Task" - echo the variable value.


------------------------------------------------------------------------------------------------------------

Thursday, November 21, 2013

SSIS:BI Env.:Performance Tuning SSIS


SQL Server Integration Services SSIS Best Practices



Refer Source Video: Free SQL Training - Performance Tuning SSIS

Graphically deselecting column does not take out column, in SQL query specify the columns.Its a Client Side filter, so it pulls all data across network.

oledb source: Table or View
SQL Command:
Select * From tablename
                        [ instead for performance: Select col1,col2 from tablename]

Data Flow Task
Source - Advanced editor - Input Output Properties - column:Fastparse

For each individual column the above step need to be repeated.Fastparse will trust data being correct and load , it improves performance a lot but if any row bad then it will fail and need to rollback.

--
Refer Source Video:SSIS & SSAS BI Platform Performance Tuning

OLTP -(SSIS) or TSQL ->DW->SSAS cube<-Reports(Share point Portal,Excel etc.)

SSAS Cube in separate server  for cube to process faster.
Using TSQL fine , not necessarily only SSIS component.
Source - 'FastParse' can be used if  data is correct.
Packet size- SQL Server 2008 ( 2 types compressions: datatype ( Select data fast , Insert slow: integer takes 1 byte instead of 4 bytes) Page Compression:WinZip does for files.


Wednesday, November 13, 2013

SQL:SQL FULL OUTER JOIN-SQL UNION -- SQL UNION ALL

Refer the Source Video: SQL Server join :- Inner join,Left join,Right join and full outer join

Full outer join=Result of (Inner join (common records) + Left Outer Join + Right Outer Join)


Source: SQL FULL OUTER JOIN Keyword

SELECT Customers.CustomerName, Orders.OrderID
FROM Customers
FULL OUTER JOIN Orders
ON Customers.CustomerID=Orders.CustomerID
ORDER BY Customers.CustomerName;

Note: The FULL OUTER JOIN keyword returns all the rows from the left table (Customers), and all the rows from the right table (Orders). If there are rows in "Customers" that do not have matches in "Orders", or if there are rows in "Orders" that do not have matches in "Customers", those rows will be listed as well.


Source: SQL UNION -- SQL UNION ALL 

SELECT City FROM Customers
UNION
SELECT City FROM Suppliers
ORDER BY City;

Note: UNION cannot be used to list ALL cities from the two tables. If several customers and suppliers share the same city, each city will only be listed once. UNION selects only distinct values. Use UNION ALL to also select duplicate values!

SSIS:Performance Tuning


SQL Server Integration Services SSIS Best Practices



Refer the Source Video:SSIS Performance Tuning

FULL RECOVERY - lot of  logging done

DW project - generally Simple Recovery Mode (minimal logging)


Simple Recovery it loads lot of rows in sec.

Control Flow  Data Flow

Control Flow:

              Create DB 
                  {notes:Create DB ( Execute SQL Task:SQL Statement )}
                   {notes: ALTER DATABASE [DBname] SET RECOVERY FULL

              Create Table

Data Flow:


Flat File Source ( 6,000,000 rows)

OLE DB Destination

notes:
CREATE DATABASE (Transact-SQL)

  Create DB ( Execute SQL Task:SQL Statement )

USE [master]
GO
IF DB_ID(DBNAME) IS NOT NULL
BEGIN
ALTER DATABASE [DBNAME] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
DROP DATABASE DBNAME;
END
GO

CREATE DATABASE [DBNAME]
CONTAINMENT = NONE
ON PRIMARY
(NAME = N'DBNAME', FILENAME = N'C:\ProgramFiles\Microsoft SQL Server\MS SQL11.MSSQLSERVER\MSSQL\DATA\DBNAME.mdf', SIZE =10 MB, MAXSIZE = UNLIMITED, FILEGROWTH = 1 MB)

LOG ON
( NAME = N'DBNAME_log', FILENAME = N' C:\ProgramFiles\Microsoft SQL Server\MS SQL11.MSSQLSERVER\MSSQL\DATA\DBNAME.ldf', SIZE = 2 MB, MAXSIZE = 2000GB, FILEGROWTH = 10%)

GO
GO
ALTER DATABASE [DBNAME] SET RECOVERY FULL
-----------------------------------------------------------------------------------------------------------

Source: SSIS Performance Tuning Simple Load Part 2 Database Growth SSIS

In the CREATE DATABASE  statement if  the FILEGROWTH  is increased  say 20MB or 100MB , performance improves.

-----------------------------------------------------------------------------------------------------------
Source:SSIS Performance Tuning Simple Load Part 3 Initial Database Size 

In previous video using the "DBGrowth"  which refer to "FILEGROWTH" the performance improved.The table that being loaded is below  the "FILEGROWTH" so it will not impart on performance.

By changing the "Initial Database Size" which refers to "SIZE"  , it will change the performance. 

-----------------------------------------------------------------------------------------------------------
 Source:SSIS Performance Tuning Simple Load Part 4 BI vs Enterprise Edition

In Enterprise Edition its faster.  
-----------------------------------------------------------------------------------------------------------
Source:SSIS Performance Tuning Simple Load Part 5 Commit Size

Data Flow:

OLE DB Destination
-----------
OLE DB Destination Editor:

             Configure the properties used to insert data into a relational database using an OLE DB provider.

[Connection Manager
Mappings
Error Output]

Specify an OLE DB Connection manager, a data source, or a data source view, and select the data access mode.If using the SQL Command access mode, specify the SQL Command either by typing
the query or by using Query Builder. For Fast - load access, set the table update options.

OLE DB Connection manager:
oledb-LoadDatabase

Data access mode:
Table or View - fast load

Name of  the table or the view:
[dbo].[Tablename..]

                        []Keep Identity [*]Table lock
[]Keep nulls []Check Constraints

Rows. per batch:

Maximum insert commit size: 0 ( changed from a higher number to 0) ( it means 1 transaction)[1 transaction: it will commit all rows, in 1 time{Logsize,TempDB size will be  bigger]
[Caution: Table having "Clustered Index"  won't work.][Only for heap table(having no indexes)]

View Existing Data
----
~Source:Making Fast Load really fast on clustered indexed tables with SSIS

Moreover, as using MICS parameter leads to a dramatic performance degradation, I think that the OleDb destination adapter should issue a warning if both ORDER and MICS parameter are set so to make the programmer think twice before using them together.

~Source:Batch Sizes, Fast Load, Commit Size And The OLE DB Destination

Rows per batch:
This setting affects the performance and commit semantics of the SQL engine, but doesn’t affect things on the SSIS side. Lowering this value does not affect the number of times SSIS commits data to the server.

Maximum insert commit size:
Enlisting the Data Flow task in a transaction also has the same effect as setting FastLoadMaxInsertCommitSize  = 0.  All rows will be committed or rolled back at the end of the Data Flow Task.

Beware
Setting the Maximum insert commit size property to a low value will cause a lot of overhead on the destination as the rows will be committed much more often.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SSIS Performance Tuning Simple Load Part 6 Packet Size SSIS 2012 Video Example

Ole Db -right click -  All - see  "Packet Size:0" - by default it means 4kb. Change it to 32KB which is maximum, if the value disappears then its on the "Connection" , choose the "project.param" , choose connection
string and specify  "Packet Size" 

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Source: SSIS Performance Tuning Simple Load Part 7 Row vs Page Compression SSIS 2012 Video Example

Connect to Server

Server type: Database Engine

Choose the Database->Table(right click "Storage" - "Manage Compression" - change Compression type "Page"

Next screen "Data Compression Wizard" , Run immediately ,

Task Manager - Look at "Performance", "Write Speed" is less , "Read speed" as usual.

Conclusion:Compression "Slows down" the load into "heap" table. 

Observation:"Row" Compression is bit faster , "Write Speed" is little faster , "Read speed" little better

But any how "Row" Compression is slower.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Source: SSIS Performance Tuning Simple Load Part 8 Balanced Data Distributor SSIS 2012 Video Example

Balanced Data Distributor , does parallel processing.It works like "Conditional Split" , but its more efficient.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Source:SSIS ole db vs SQL Server Destination Simple Performance Test


OLEDB destination is faster and its recommened but in this example SQL Server Destination is faster.
----------------------------------------------------------------------------------------------------------------------------------

Friday, November 8, 2013

SSIS:Conditional Split Transformation

Refer Source Video: SSIS Conditional Split Transformation Part 1 Video Example SSIS 2012

2005,2008,2008 R2, 2012 - same. No much difference.
DataFlow

OLEDB DataSource:

SQL Command: query ( UNION ALL  of  two resultset)

Derieved Column: To view the values , "Condition" , column define the condition for spliting. The "Order" of  the condition important, accordingly it will execute.

Why "Order" is important, see in the below video.

Refer Source Video:SSIS Conditional Split IS NOT NULL Part 3 Video Example SSIS 201

If you have NULL  value then the Conditional Split Transformation , will stop. If  one adds like ISNULL(columnname) as the codition , it still will not work and other condition it will not run until the "Order" the ISNULL having condition is first.

Refer Source Video:SSIS Conditional Split Default Output Name Part 2 Video Example SSIS

"Conditional Split Transformation" Editor, the rows that do not meet any condition defined in the condition criteria will be lost, to capture it , in "Default Output name:" give a name and in Data Flow add a Derived Column & add , so those rows are captured.

Refer Source Video:SSIS Conditional Split Configure Error Output Part 4 Video Example

If  you have NULL value or for any condition you want those row to fall into another category then in the "Conditional Split Transformation" , there is a "Red Arrow" in that you specify the "Redirect" output of conditions , thats the same you see in "Configure Error Output", so using a 'Derieved Column" , one can see the error rows.

------------------------------------------------------------------------------------------------------------
Refer Source Video:Conditional Split Transformation In SSIS

If  in "Condition Split Transformation" one want to split data and put into different sources like one to "Flat file" and other set to  any other database.

------------------------------------------------------------------------------------------------------------

Wednesday, October 16, 2013

SSIS:Logging

Refer the Source Video: SSIS:Logging

Right click in Package - Logging

Sql Task

Containers: Flexibility to choose a task
                  ( allows task wise track error messages.)

Logs     Details(Event wise error messages: On Error, OnPostExecute,OnPreExecute)

Provider Type:
SSIS log  Provider for windows
SSIS log  Provider for Text files (.txt)
SSIS log  Provider for XML files (.xml)
SSIS log  Provider for SQL Server
SSIS log  Provider for SQL Server Profiler (.trc) {Only for 32bit, change in right clicking on Project Explorer,,)

Mostly used:Windows & SQL Server one

Windows see in Windows Event Viewer:
Select * from SYSSSISLog 

Tuesday, October 15, 2013

SSIS:Deployment using Project Deployment Model

Refer Source Video:SSIS 2012 Deployment using Project Deployment Model Video

--------------------------------
Prior:How to create a Catalog?
----------------------------------
Project Deployment Model  is new in 2012:
Folder file names: Package,Project.params,SSIS Project Deployment Model,SSIS Type Integration Services project file
----
Method1:Right click on project-choose Deploy
Server:Database Engine Server Name ( where SSISDB Catalog is their).
                        Path:SSISDB\Test

Copy the commandline parameter to Notepad.
The project is deployed to Database Engine(--verify)
Microsoft SQL Server Management Studio:
  Integration Services Catalogs-SSISDB-Test-Projects-SSIS Project Deploment Model-Packages-Package.dtsx
  SQL Server Agent
--
After running it the Bin\Deployment \ folder is created with file SSIS Project Deploment Model.
-----
Method2: Right click on Project Explorer->Build
Prior to choosing "Build" remove the Bin & Object folder

It will create Bin\Development , creates SSIS Project Deploment Model (Integration Services Deployment File) [Instead of running this , move to another Server & Run from their]
just double click the file it will run the wizard.( Same as Method 1 wizard)

Method3: Running from Commandline (---Notepad)
cd %ProgramFiles%\Microsoft SQL Server\..\Binn
isdeploymentwizard.exe  whatever copied for commandline

--Project deployment method whole Project is deployed not package wise.If any one modifying a package , make sure that latest one is compiled.