Excel User Group
Microsoft Excel blogs, forums, files. Read, ask questions, provide answers.

Importing text files, best way? VBA, Access, SQL??

Latest post Mon, Jun 28 2010 5:26 AM by cferart. 29 replies.
  • Mon, Jun 7 2010 6:03 AM

    • cferart
    • Top 75 Contributor
    • Joined on Mon, Jun 7 2010
    • Posts 14
    • Points 214

    Importing text files, best way? VBA, Access, SQL??

    Hi, I am on a steep learning curve with Excel... I have some great success with importing and joining multiple .csv files via SQL AND QUERY.

    So I wondered if I can do something like this with .txt files? Its not a once off situation, so I would like to automate it so others can do it.

     

    Firstly any ideas, or advice about how to start? the .txt files are comma delimited and I can import one by one via the Data import external data process and then clean them up. However this is a tedious process.  

     

    One Idea I had was importing into Access (I am a complete beginner here) perhaps easier to do this in Access than in Excel?

    As you can see I have lots of ideas??? Happy to take on board any suggestions to investigate further!

     

    many thanks

    Catherine

     

     

    Filed under:
    • Post Points: 21
  • Tue, Jun 8 2010 12:36 PM In reply to

    • Nick Hodge
    • Top 10 Contributor
    • Joined on Sun, Dec 23 2007
    • Southampton
    • Posts 543
    • Points 9,287
    • MVP

    Re: Importing text files, best way? VBA, Access, SQL??

    Catherine

    I would turn on the macro recorder while doing you import and clean. This way you will get some VBA code. You will then need to tidy that up and use some variables to look for a different file name each time, etc. If you get that bit done and post back I'm sure we'll be able to help cleaning the code that results

    Regards
    Nick Hodge
    Microsoft MVP, Excel
    Southampton, UK

    • Post Points: 36
  • Tue, Jun 8 2010 7:50 PM In reply to

    • cferart
    • Top 75 Contributor
    • Joined on Mon, Jun 7 2010
    • Posts 14
    • Points 214

    Re: Importing text files, best way? VBA, Access, SQL??

    Thanks Nick

    I will give this a go, and I will post the code as I will need assistance to make it work for different files and append the data.

     

    cheers

    Catherine

    • Post Points: 5
  • Wed, Jun 9 2010 8:29 PM In reply to

    • cferart
    • Top 75 Contributor
    • Joined on Mon, Jun 7 2010
    • Posts 14
    • Points 214

    Re: Importing text files, best way? VBA, Access, SQL??

    Hi Nick

    I have recorded 2 macros. I am very very new at Macros and VBA and would appreciate help in working out how to make it prompt for File

    1.Imports text file and gets rid of unnecessary columns

    Sub importtextfile()
    '
    ' importtextfile Macro
    ' Macro recorded 10/06/2010 by Catherine
    '

    '
        With ActiveSheet.QueryTables.Add(Connection:= _
            "TEXT;C:\Documents and Settings\e06421\Desktop\abc.txt" _
            , Destination:=Range("A1"))
            .Name = "abc"
            .FieldNames = True
            .RowNumbers = False
            .FillAdjacentFormulas = False
            .PreserveFormatting = True
            .RefreshOnFileOpen = False
            .RefreshStyle = xlInsertDeleteCells
            .SavePassword = False
            .SaveData = True
            .AdjustColumnWidth = True
            .RefreshPeriod = 0
            .TextFilePromptOnRefresh = False
            .TextFilePlatform = -535
            .TextFileStartRow = 1
            .TextFileParseType = xlDelimited
            .TextFileTextQualifier = xlTextQualifierDoubleQuote
            .TextFileConsecutiveDelimiter = False
            .TextFileTabDelimiter = True
            .TextFileSemicolonDelimiter = False
            .TextFileCommaDelimiter = False
            .TextFileSpaceDelimiter = False
            .TextFileColumnDataTypes = Array(1, 9, 9, 9, 9, 9, 1, 9, 1, 1, 1, 1, 9, 9, 9, 9, 9, 9, 9, 9, 9, _
            9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 1, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9 _
            , 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, _
            9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9 _
            , 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, _
            9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9)
            .TextFileTrailingMinusNumbers = True
            .Refresh BackgroundQuery:=False
        End With

        Columns("H:BK").Select
        Selection.Delete Shift:=xlToLeft
        Range("J20").Select
    '
    End Sub

    2. Copies imported data to sheet 1

    Sub Appendtxtdata()
    '
    ' Appendtxtdata Macro
    ' Macro recorded 10/06/2010 by Catherine
    '

    '
        Range("A2").Select
        Range(Selection, Selection.End(xlToRight)).Select
        Range(Selection, Selection.End(xlDown)).Select
        Selection.Cut
        Sheets("Sheet1").Select
        Range("A13").Select
    End Sub

     

    Any help appreciated

    thansks

    Filed under: , ,
    • Post Points: 21
  • Thu, Jun 10 2010 3:57 AM In reply to

    • Nick Hodge
    • Top 10 Contributor
    • Joined on Sun, Dec 23 2007
    • Southampton
    • Posts 543
    • Points 9,287
    • MVP

    Re: Importing text files, best way? VBA, Access, SQL??

    Catherine

    I have commented these code lines (prefixed with an '. these do not run they are just notes.

    I have shortened the import code, removing the defaults the macro recorder does which are not necessary. I do not understand why you are deleting the columns as the 9's in the array mean they should not import at all??? (I have left the column deleting out therefore but could easily put it back.

    I am taking a new sheet (not sheet1) and copying the code onto there (I could easily rename the sheet tab to sheet1 if you wanted?). I then delete the original import sheet. which as you were cutting the data seemed Ok anyhow.

    If you have questions on the code or wish it modified then shout. It should now present you with a dialog to choose the file (restricted to *.txt files) and go from there. We could copy the code of the second macro into the first two and do everything in one fell swoop if that would help.

    Anyhow, here's the code

    Option Explicit

    Sub importtextfile()
    '
    ' importtextfile Macro
    ' Macro recorded 10/06/2010 by Catherine
    '

    'Set aside a variable for the name of the file
    Dim sFileName As String

    'Collect the file name. This does nothing, just collects the text of the filename
    sFileName = Application.GetOpenFilename("Text Files (*.txt), *.txt")

    'this just checks if someone pressed cancel on the dialog
    If sFileName = "False" Then Exit Sub

    'This is your recorded code with the variable in place of teh hard-coded name
    'and the defaults taken out (not needed)
    'all the 9's are columns which shouldn't import.
        With ActiveSheet.QueryTables.Add(Connection:=sFileName _
            , Destination:=Range("A1"))
            .Name = "abc"
            .TextFileParseType = xlDelimited
            .TextFileTabDelimiter = True
            .TextFileColumnDataTypes = Array(1, 9, 9, 9, 9, 9, 1, 9, 1, 1, 1, 1, 9, 9, 9, 9, 9, 9, 9, 9, 9, _
            9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 1, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9 _
            , 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, _
            9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9 _
            , 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, _
            9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9)
        End With
    End Sub


    Sub Appendtxtdata()
    '
    ' Appendtxtdata Macro
    ' Macro recorded 10/06/2010 by Catherine
    '
    'Resereves space for variables
    Dim lLastRow As Long, lLastColumn As Long
    Dim wksNew As Worksheet, wksCurrent As Worksheet

    'Commissions and error handlers should anything fail
    On Error GoTo ErrHandler

    'With construct saves typing Object (in this case Application) on each line
    'This statement stops the screen updating (flashing) and any warnings popping up
    With Application
        .DisplayAlerts = False
        .ScreenUpdating = False
    End With

    'If you have an object variable (see above) you need to set it. Now
    'I can just refer to these variables and they will always point to
    'that object, for example no matter what worksheet is active
    Set wksCurrent = ActiveSheet

    'This one adds a new sheet. I don't need to know the name as it will always be
    'called sksNew to me in this code
    Set wksNew = Worksheets.Add

    'finds the last row and column in the data and then copies that range
    'and pastes it in one operation and deletes the original sheet
    With wksCurrent
        lLastRow = .Cells(.Rows.Count, "A").End(xlUp).Row
        lLastColumn = .Cells(1, .Columns.Count).End(xlToLeft).Column
        .Range(.Cells(1, 1), .Cells(lLastRow, lLastColumn)).Copy Destination:=wksNew.Range("A1")
        .Delete
    End With

    'Switches screen updating and alerts back on
    With Application
        .ScreenUpdating = True
        .DisplayAlerts = True
    End With

    'Exits the routine if successful otherwise it would run the error handler as
    'the next line of code
    Exit Sub

    'Error handler should anything go wrong which pops a message box
    'and resets stuff
    ErrHandler:
    MsgBox "Somthing went wrong", vbInformation + vbOKOnly, "Error"
    With Application
        .ScreenUpdating = True
        .DisplayAlerts = True
    End With

    End Sub

     

    Regards
    Nick Hodge
    Microsoft MVP, Excel
    Southampton, UK

    • Post Points: 21
  • Thu, Jun 10 2010 4:03 AM In reply to

    • cferart
    • Top 75 Contributor
    • Joined on Mon, Jun 7 2010
    • Posts 14
    • Points 214

    Re: Importing text files, best way? VBA, Access, SQL??

    thanks Nick

    I was reading a book on VBA on the way home via the train, to start gaining an understanding... early days!

     

    I really appreciate you taking the time to help me with this... I am so amazed by what Excel can do... and how little we use all its features...

    I will run this at work tommorrow and let you know...

     

    Cheers

    Catherine

     

    • Post Points: 5
  • Thu, Jun 10 2010 7:42 PM In reply to

    • cferart
    • Top 75 Contributor
    • Joined on Mon, Jun 7 2010
    • Posts 14
    • Points 214

    Re: Importing text files, best way? VBA, Access, SQL??

    Hi Nick

    I am getting an error "Application-defined or objective-defined"

    at the point

        With ActiveSheet.QueryTables.Add(Connection:=sFileName _
            , Destination:=Range("A1"))

    any ideas?

     

    many thanks

    Catherine

    • Post Points: 21
  • Fri, Jun 11 2010 1:51 AM In reply to

    • Nick Hodge
    • Top 10 Contributor
    • Joined on Sun, Dec 23 2007
    • Southampton
    • Posts 543
    • Points 9,287
    • MVP

    Re: Importing text files, best way? VBA, Access, SQL??

    Catherine

    I'm so sorry, in my haste to get off the train yesterday I cut the connection qualifier out of the file name to (TEXT;), paste this line in over the one you have

        With ActiveSheet.QueryTables.Add(Connection:="TEXT;" & sFileName, _
             Destination:=Range("A1"))

    Regards
    Nick Hodge
    Microsoft MVP, Excel
    Southampton, UK

    • Post Points: 21
  • Fri, Jun 11 2010 5:58 AM In reply to

    • cferart
    • Top 75 Contributor
    • Joined on Mon, Jun 7 2010
    • Posts 14
    • Points 214

    Re: Importing text files, best way? VBA, Access, SQL??

    Thanks, Nick I didn't have much time at work today, but i did do a couple of quick tries after replacing this text, but while it did not go into debug, nothing appeared to happen.

    Would the connection process take some time? I used f8 to step through to the end, but nothing changed on the sheet? 

    I am working tommorrow morning Sat (australia time) and will have another look.. if you have any ideas happy and thank ful to hear them.

    cheers

    Catherine

    • Post Points: 21
  • Fri, Jun 11 2010 1:05 PM In reply to

    • Nick Hodge
    • Top 10 Contributor
    • Joined on Sun, Dec 23 2007
    • Southampton
    • Posts 543
    • Points 9,287
    • MVP

    Re: Importing text files, best way? VBA, Access, SQL??

    Catherine

    That's wierd I get that too although the string matches the syntax in the hard coded version. Could you attach or send me a small text file that matches the spec. nick_hodge at zen dot co dot uk

    Regards
    Nick Hodge
    Microsoft MVP, Excel
    Southampton, UK

    • Post Points: 5
  • Mon, Jun 14 2010 4:00 AM In reply to

    • Nick Hodge
    • Top 10 Contributor
    • Joined on Sun, Dec 23 2007
    • Southampton
    • Posts 543
    • Points 9,287
    • MVP

    Re: Importing text files, best way? VBA, Access, SQL??

    Catherine

    Geez. I need to go back to school. In error I had removed the Refresh command at the bottom. That is why no data came through. The code below should now work. I have set your ISBN to come through as text ('2'), in the array as Excel sees it as a number and being so long changes it to scientific notation. I suspect you would never need it as a number so hence the change.

    Here is the code

    Sub importtextfile()
    '
    ' importtextfile Macro
    ' Macro recorded 10/06/2010 by Catherine
    '

    'Set aside a variable for the name of the file
    Dim sFileName As String

    'Collect the file name. This does nothing, just collects the text of the filename
    sFileName = Application.GetOpenFilename("Text Files (*.txt), *.txt")

    'this just checks if someone pressed cancel on the dialog
    If sFileName = "False" Then Exit Sub

    'This is your recorded code with the variable in place of teh hard-coded name
    'and the defaults taken out (not needed)
    'all the 9's are columns which shouldn't import.
            With ActiveSheet.QueryTables.Add(Connection:= _
            "TEXT;" & sFileName _
            , Destination:=Range("A1"))
            .TextFileTabDelimiter = True
            .TextFileColumnDataTypes = Array(1, 9, 9, 9, 9, 9, 1, 9, 1, 1, 2, 1, 9, 9, 9, 9, 9, 9, 9, 9, 9, _
            9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 1, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9 _
            , 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, _
            9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9 _
            , 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, _
            9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9)
            .Refresh BackgroundQuery:=False
        End With
    End Sub

    Let me know if it works and if you need any further developments. This will allow selection of the file from a dialog now

    Regards
    Nick Hodge
    Microsoft MVP, Excel
    Southampton, UK

    • Post Points: 37
  • Mon, Jun 14 2010 6:20 AM In reply to

    Re: Importing text files, best way? VBA, Access, SQL??

    Why bother with the query, why not just open the text files and do a Text to Columns on them all directly?

    Regards

    Bob

    • Post Points: 21
  • Mon, Jun 14 2010 3:07 PM In reply to

    • Nick Hodge
    • Top 10 Contributor
    • Joined on Sun, Dec 23 2007
    • Southampton
    • Posts 543
    • Points 9,287
    • MVP

    Re: Importing text files, best way? VBA, Access, SQL??

    Bob

    Is that not as broad as it's long?

    Catherine. I would suggest you put this in your personal.xls(m). That way it will be available everytime Excel opens to all open workbooks. You could then give it a shortcut key or button maybe?

     

    Let us know if you want help with this

    Regards
    Nick Hodge
    Microsoft MVP, Excel
    Southampton, UK

    • Post Points: 21
  • Mon, Jun 14 2010 3:40 PM In reply to

    Re: Importing text files, best way? VBA, Access, SQL??

    In effect yes, but it seems logically much more straight-forward than querying (but then, you are query obsessed, and I NEVER use it Stick out tongue).

    Regards

    Bob

    • Post Points: 5
  • Mon, Jun 14 2010 7:59 PM In reply to

    • cferart
    • Top 75 Contributor
    • Joined on Mon, Jun 7 2010
    • Posts 14
    • Points 214

    Re: Importing text files, best way? VBA, Access, SQL??

     

    This works brilliantly! thank you so much..

    I still need help with the appending data, and I may not have made this clear.

    At the moment it creates a new sheet and places data there and then deletes existing sheet. However what is really needed is the ability to add the data at the end of any existing data the user may have.

    So how does the VBA know which sheet to append data? User enters sheet name?

    I am not sure as part of the import text operation I can tell it in the Destination to goto last row ? Or this maybe too complex in this statement.

    Any ideas of this, hopefully, last piece of the puzzle?

     

    thanks

    Catherine

    • Post Points: 5
Page 1 of 2 (30 items) 1 2 Next > | RSS
Copyright Excel User Group and the relevant contributors, 2010. All Trade Marks acknowledged. This site is a peer-to-peer site and NOT affiliated in any way to Microsoft Corp. All rights reserved.