PDA

View Full Version : copy and tar problem



albumns
20-Sep-2011, 07:46
Hi:

I found there is some problem with SLE problem: each time I am
copying large files into removable disk or I am tar some big files, the
whole system became very slow: almost all the program and windows stuck
there..... No matter how well (I get a 24 CPU (3.6 GHZ), 32 GB memory
workstation) is my CPU or memory is, the problem always happen. I don't
install beagle and compiz in my system.

How to solve this problem ?

THX


--
albumns
------------------------------------------------------------------------
albumns's Profile: http://forums.novell.com/member.php?userid=48883
View this thread: http://forums.novell.com/showthread.php?t=444967

malcolmlewis
20-Sep-2011, 12:22
On Tue, 20 Sep 2011 06:46:04 GMT
albumns <albumns@no-mx.forums.novell.com> wrote:

>
> Hi:
>
> I found there is some problem with SLE problem: each time I am
> copying large files into removable disk or I am tar some big files,
> the whole system became very slow: almost all the program and windows
> stuck there..... No matter how well (I get a 24 CPU (3.6 GHZ), 32 GB
> memory workstation) is my CPU or memory is, the problem always
> happen. I don't install beagle and compiz in my system.
>
> How to solve this problem ?
>
> THX
>
>
Hi
Sounds like I/O rather than CPU/RAM. How are you transferring to
removable disk, USB eSATA? How fast are the hard drives in your system,
10K RPM or SSD?

--
Cheers Malcolm °¿° (Linux Counter #276890)
openSUSE 11.4 (x86_64) Kernel 2.6.37.6-0.7-desktop
up 7 days 14:19, 3 users, load average: 0.08, 0.16, 0.19
GPU GeForce 8600 GTS Silent - Driver Version: 280.13

ab
20-Sep-2011, 15:19
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Exactly. The main bottleneck of almost every system in the world is
I/O. When your system is busy using the disk, especially when it is
reading from it (to get the files) and writing to it (to create the tar)
at the same time your system will seem slow because of that bottleneck
no matter how much RAM or how many CPUs you have.

When you're sitting at a stoplight (waiting on something else) it
doesn't matter how fast your car goes or how much your pickup can carry;
the stop light is going to keep you from getting anything done.

Good luck.


- --
Want to yell at me in person?
Come to BrainShare 2011 in October: http://tinyurl.com/brainshare2011
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJOeKDSAAoJEF+XTK08PnB5U38QAMLQFsIVYa CASpg/ZbNPDGmL
8kNCyW5Z3RRkTrBXm4VD4nu1jjDE5UV4/AAwagE6X4uEIFIOPu0IPlgW/5jIWY7f
pVJLcxVUxq2FxgFvE7wcwImhYOBx8EecEdz+fl5bH4vEQErsrs 6pKKABXXivKhoM
SHzLuODunOLHJhbrhjkIWbESlgMS8PfQtjLTccqaYQRsZ+5naQ 12jiuqYS1wXPNu
IhwhpI+2rVNbTSnx530oDFRkcPhAjvzoK85lfd07ZEPzNJLpWE u9DWwWJ2ci0UNO
oVVIkjA1OXV/24zWUcJWeAoTw+iLkuu0niOfq0U/ODXCvaIRUyRclwA+UWa2KeIx
ZFwIhFcs6oSQaR71dPHKm0zwFoSsjToFgcMAEFXds6E3ntSn7T AdyLuHwxyBzgVc
gqqFgBSKNus1NsUQjA7+awmAPxtOnTEqb1qPLi+9Q6C3+1h59e f+88tQApiAvWDU
hrIgWAvPLZ//8l/ZEj+HXG/pepsiiKNF0fOlAmjrztS8N0+bKW5at6a/CCiyqLkP
gPFSaGW8Vplo0XlguoA7OosFq5OgjQZ8Fn5XZheLEaUOI4uI65/MZlxAD3YhakAf
0q2wfCd+5S7735VVSty0z9Qtcf9CiDr6wf7OXMOPZQtNYdFOJU kzvt+0a9WIqSRT
ccpIq8TMcELKObkGwXJv
=8NEc
-----END PGP SIGNATURE-----

albumns
20-Sep-2011, 18:16
malcolmlewis;2138888 Wrote:
> On Tue, 20 Sep 2011 06:46:04 GMT
> albumns <albumns@no-mx.forums.novell.com> wrote:
>
> >
> > Hi:
> >
> > I found there is some problem with SLE problem: each time I am
> > copying large files into removable disk or I am tar some big files,
> > the whole system became very slow: almost all the program and windows
> > stuck there..... No matter how well (I get a 24 CPU (3.6 GHZ), 32 GB
> > memory workstation) is my CPU or memory is, the problem always
> > happen. I don't install beagle and compiz in my system.
> >
> > How to solve this problem ?
> >
> > THX
> >
> >
> Hi
> Sounds like I/O rather than CPU/RAM. How are you transferring to
> removable disk, USB eSATA? How fast are the hard drives in your system,
> 10K RPM or SSD?
>
> --
> Cheers Malcolm °¿° (Linux Counter #276890)
> openSUSE 11.4 (x86_64) Kernel 2.6.37.6-0.7-desktop
> up 7 days 14:19, 3 users, load average: 0.08, 0.16, 0.19
> GPU GeForce 8600 GTS Silent - Driver Version: 280.13


my disk is 7200 rpm. I use USB removable disk for copying files. But I
don't understand even I tar the files or untar files in local machine,
the whole system becomes very slow. Is there any methods to solve this
problem?

THX


--
albumns
------------------------------------------------------------------------
albumns's Profile: http://forums.novell.com/member.php?userid=48883
View this thread: http://forums.novell.com/showthread.php?t=444967

albumns
20-Sep-2011, 18:16
ab;2138966 Wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Exactly. The main bottleneck of almost every system in the world is
> I/O. When your system is busy using the disk, especially when it is
> reading from it (to get the files) and writing to it (to create the
> tar)
> at the same time your system will seem slow because of that bottleneck
> no matter how much RAM or how many CPUs you have.
>
> When you're sitting at a stoplight (waiting on something else) it
> doesn't matter how fast your car goes or how much your pickup can
> carry;
> the stop light is going to keep you from getting anything done.
>
> Good luck.
>
>
> - --
> Want to yell at me in person?
> Come to BrainShare 2011 in October: http://tinyurl.com/brainshare2011
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.15 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - 'Enigmail: A simple interface for
> OpenPGP email security' (http://enigmail.mozdev.org/)
>
> iQIcBAEBAgAGBQJOeKDSAAoJEF+XTK08PnB5U38QAMLQFsIVYa CASpg/ZbNPDGmL
> 8kNCyW5Z3RRkTrBXm4VD4nu1jjDE5UV4/AAwagE6X4uEIFIOPu0IPlgW/5jIWY7f
> pVJLcxVUxq2FxgFvE7wcwImhYOBx8EecEdz+fl5bH4vEQErsrs 6pKKABXXivKhoM
> SHzLuODunOLHJhbrhjkIWbESlgMS8PfQtjLTccqaYQRsZ+5naQ 12jiuqYS1wXPNu
> IhwhpI+2rVNbTSnx530oDFRkcPhAjvzoK85lfd07ZEPzNJLpWE u9DWwWJ2ci0UNO
> oVVIkjA1OXV/24zWUcJWeAoTw+iLkuu0niOfq0U/ODXCvaIRUyRclwA+UWa2KeIx
> ZFwIhFcs6oSQaR71dPHKm0zwFoSsjToFgcMAEFXds6E3ntSn7T AdyLuHwxyBzgVc
> gqqFgBSKNus1NsUQjA7+awmAPxtOnTEqb1qPLi+9Q6C3+1h59e f+88tQApiAvWDU
> hrIgWAvPLZ//8l/ZEj+HXG/pepsiiKNF0fOlAmjrztS8N0+bKW5at6a/CCiyqLkP
> gPFSaGW8Vplo0XlguoA7OosFq5OgjQZ8Fn5XZheLEaUOI4uI65/MZlxAD3YhakAf
> 0q2wfCd+5S7735VVSty0z9Qtcf9CiDr6wf7OXMOPZQtNYdFOJU kzvt+0a9WIqSRT
> ccpIq8TMcELKObkGwXJv
> =8NEc
> -----END PGP SIGNATURE-----




*well, but why this doesn't happen in windows.....?*


--
albumns
------------------------------------------------------------------------
albumns's Profile: http://forums.novell.com/member.php?userid=48883
View this thread: http://forums.novell.com/showthread.php?t=444967

ab
20-Sep-2011, 18:50
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

It does, but you don't notice it because windows is slow all of the
time. ;-)

If you really think it doesn't happen in windows you're really not
looking. I/O is a constant in the computing world... it's inherent with
all kinds of things and especially disks with moving (spinning) parts.

Good luck.



- --
Want to yell at me in person?
Come to BrainShare 2011 in October: http://tinyurl.com/brainshare2011
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJOeNI7AAoJEF+XTK08PnB5Uw8QALC9Nrh4dO gxDjn3+gbafk57
zGvFrKmu5QlkAnoLF8akpEcx/tfBoBOUXjfVkQ+LmGlw5MbR9RNmp+5TVEmfdz+A
hA0zCIeb+BSSB545E+DeFAGERdtEZXNlaB3lgeHK5AF5u2foc3 K743dVLHaFY+28
MLNxyIBPf3r80MAa6jjILb8FThZzP5NxFmZWSAV+ZntApW7KuS UxB93uCAhUKi/P
VoeEVyrDkUPlrHVoLZB/Yv4MZJbv6iTkOO+ULiXjhcGOTdRWrFrEWCQQGgisKZB0
CoeeA4BlZcRoMeiJsypHIeAYjc00pHUJUFhOpU9e1KoO5peRg2 +rV2exaPkfdAhz
4LbCazfKrdqAzsFm994oOCt5LRqpfy4RH8nukcqKEZUrxkXqEk SUdmniaYw60wLx
yTJRvF3KA8DkiFXdteJRcxOjCvpvEmpCynCQqnFnMHPlFwXgHg BRqG0KQYQrsZq+
cbHpNR0SG30/GMUcpTY+GexBjH/5OR40YH1yVHG+urvzu66ZWGNAe2eHV1GIbsBj
+D2M4xB59oNADo4B7HVXakaog9KEDLdv+aVtb5g22uObbjNYVZ ZflO/UymDCM7WN
S080MjsO9a4gtdskCVuwJ2lHOlFKzZilc4jLJ9JSiPW8G4Su/4o1MgcLbyloUDQG
2RUN4hsZmqa2s01PyyWk
=ekw/
-----END PGP SIGNATURE-----

ab
20-Sep-2011, 18:51
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

How are you objectively measuring the slowness of the system
before/during/after the tar operation?

Good luck.


- --
Want to yell at me in person?
Come to BrainShare 2011 in October: http://tinyurl.com/brainshare2011
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJOeNJ2AAoJEF+XTK08PnB5i3oQAIU59QCGeb 78Ytuphp0jzqHP
jye1lgAHK+kSW5TYBDucOVBHnaTFlarp2rI5LDlo8JNNZoGOAt oI1JYs27BJk0KQ
DKtVOl93llCgRxaGS/X4VgGd+02V/R8pyJXijRgbZLH4L/ufqgo2xRyzSPNqX3D/
Yf7mHly3bcVvMtUdplM6vImYdKsUuioft8xopCBp0oKnp4NtbN YxwmObQIBGOHqZ
u3enqWqml37fHpJ+ChcrYHUdwNMhGKZV35+SUktq52pwF6a131 vu4wEI1N2jD78s
jwJTTgKEN9M564yikmFYrf10PGHSuq5oSNc833v/pr1oSwpnrciCM2Y5LieIAitr
zl6h3tM+Ml+O7NWQRy5U5w+uB+i//4HZaRjTVGe9oVrgsz9UOA+GsktaUx5xJEeT
hQxZiCFjED+t/Kio9f6g5KJpv0GZqdyDhCdhGlvSp7Gun8Z5ETQ5rtZycjqavfe e
RGKxMfwXVfi+e1ekR9859AiJYJbx94AS+SFwroXk4BsC750/C9RBF4qphhrnqicN
tSxmEymDHfIvwRE6Rz5+1Jw57e1s/tczW1sOcr7FjMuURLeuQB3LKwZJocskyiji
iu6NLmbRq9+BbHmQLqPDhXG5x7CXst2DGZaXRA9whiYXZlp0Y1 yYdCsx3yk0/A7T
mhTFfFD2M7TuMAGls+Y4
=hx1z
-----END PGP SIGNATURE-----

malcolmlewis
20-Sep-2011, 19:26
On Tue, 20 Sep 2011 17:16:01 GMT
albumns <albumns@no-mx.forums.novell.com> wrote:

>
> ab;2138966 Wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Exactly. The main bottleneck of almost every system in the world is
> > I/O. When your system is busy using the disk, especially when it is
> > reading from it (to get the files) and writing to it (to create the
> > tar)
> > at the same time your system will seem slow because of that
> > bottleneck no matter how much RAM or how many CPUs you have.
> >
> > When you're sitting at a stoplight (waiting on something else) it
> > doesn't matter how fast your car goes or how much your pickup can
> > carry;
> > the stop light is going to keep you from getting anything done.
> >
> > Good luck.
> >
> >
> > - --
> > Want to yell at me in person?
> > Come to BrainShare 2011 in October:
> > http://tinyurl.com/brainshare2011 -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v2.0.15 (GNU/Linux)
> > Comment: Using GnuPG with Mozilla - 'Enigmail: A simple interface
> > for OpenPGP email security' (http://enigmail.mozdev.org/)
> >
> > iQIcBAEBAgAGBQJOeKDSAAoJEF+XTK08PnB5U38QAMLQFsIVYa CASpg/ZbNPDGmL
> > 8kNCyW5Z3RRkTrBXm4VD4nu1jjDE5UV4/AAwagE6X4uEIFIOPu0IPlgW/5jIWY7f
> > pVJLcxVUxq2FxgFvE7wcwImhYOBx8EecEdz+fl5bH4vEQErsrs 6pKKABXXivKhoM
> > SHzLuODunOLHJhbrhjkIWbESlgMS8PfQtjLTccqaYQRsZ+5naQ 12jiuqYS1wXPNu
> > IhwhpI+2rVNbTSnx530oDFRkcPhAjvzoK85lfd07ZEPzNJLpWE u9DWwWJ2ci0UNO
> > oVVIkjA1OXV/24zWUcJWeAoTw+iLkuu0niOfq0U/ODXCvaIRUyRclwA+UWa2KeIx
> > ZFwIhFcs6oSQaR71dPHKm0zwFoSsjToFgcMAEFXds6E3ntSn7T AdyLuHwxyBzgVc
> > gqqFgBSKNus1NsUQjA7+awmAPxtOnTEqb1qPLi+9Q6C3+1h59e f+88tQApiAvWDU
> > hrIgWAvPLZ//8l/ZEj+HXG/pepsiiKNF0fOlAmjrztS8N0+bKW5at6a/CCiyqLkP
> > gPFSaGW8Vplo0XlguoA7OosFq5OgjQZ8Fn5XZheLEaUOI4uI65/MZlxAD3YhakAf
> > 0q2wfCd+5S7735VVSty0z9Qtcf9CiDr6wf7OXMOPZQtNYdFOJU kzvt+0a9WIqSRT
> > ccpIq8TMcELKObkGwXJv
> > =8NEc
> > -----END PGP SIGNATURE-----
>
>
>
>
> *well, but why this doesn't happen in windows.....?*
>
>
Hi
Maybe they change the io scheduling in the background, have no idea as
it's all subjective.

Anyway, maybe using ionice to reschedule the activity, watching the
output of the command vmstat 2 may show info.

What is the exact command your running? Are you running tar as one
operation then copying, or running tar straight to the device?

With the device plugged in can you post the output from it's dev
number eg;


hdparm -t /dev/sda


Are you using tar on windows? Remember tar is i/o bound it does a read
or a write then waits.

You need to look at something that will use all that processing
power.... pbzip2 for example.
http://compression.ca/pbzip2/
http://software.opensuse.org/search?q=pbzip2&baseproject=SUSE%3ASLE-11%3ASP1&lang=en&exclude_debug=true

--
Cheers Malcolm °¿° (Linux Counter #276890)
openSUSE 11.4 (x86_64) Kernel 2.6.37.6-0.7-desktop
up 7 days 21:09, 6 users, load average: 0.00, 0.09, 0.17
GPU GeForce 8600 GTS Silent - Driver Version: 280.13

albumns
20-Sep-2011, 19:46
ab;2139108 Wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> How are you objectively measuring the slowness of the system
> before/during/after the tar operation?
>
>


before the tar operations, everything goes well: I can do any
operations I want such as edit document. However, when I tar/untar
something big, my mouse stuck there and software windows become
white..... I can't do any operations anymore until the tar/untar
finished.

THX


--
albumns
------------------------------------------------------------------------
albumns's Profile: http://forums.novell.com/member.php?userid=48883
View this thread: http://forums.novell.com/showthread.php?t=444967

mikewillis
20-Sep-2011, 19:46
The only time I've seen behaviour like you describe was about five years
ago no a machine that was old enough to have a PATA harddisk and DMA had
somehow been disabled. The system was hopelessly unresponsive when there
was any significant disk access going on. Given the specs you mentioned
I very much doubt you have a PATA harddisk though.

What does this give you?

Code:
--------------------
$ for i in {1..5};do hdparm -t /dev/sda;done
--------------------

I get figures of around 80 MB/sec.

This sounds like an issue that could be very difficult to diagnose.
Have you considered raising a Service Request with Novell?


Hey look you can edit posts now!


--
mikewillis
------------------------------------------------------------------------
mikewillis's Profile: http://forums.novell.com/member.php?userid=7510
View this thread: http://forums.novell.com/showthread.php?t=444967

albumns
20-Sep-2011, 19:46
Honey, I don't tar or untar files all the time in my computer. Of course
I can edit posts.

Here is the output of the command:

/dev/sda:
Timing buffered disk reads: 304 MB in 3.00 seconds = 101.28 MB/sec

/dev/sda:
Timing buffered disk reads: 250 MB in 3.00 seconds = 83.32 MB/sec

/dev/sda:
Timing buffered disk reads: 306 MB in 3.01 seconds = 101.80 MB/sec

/dev/sda:
Timing buffered disk reads: 306 MB in 3.01 seconds = 101.81 MB/sec

/dev/sda:
Timing buffered disk reads: 242 MB in 3.00 seconds = 80.63 MB/sec


--
albumns
------------------------------------------------------------------------
albumns's Profile: http://forums.novell.com/member.php?userid=48883
View this thread: http://forums.novell.com/showthread.php?t=444967

friedl jan
21-Sep-2011, 09:26
Hi,
i red all replies and have some idea ;-)

Are you using tar operation on removable or on physical disk? With file
system are you using?

Do you anytime check the value "%wa = waiting operation" in the top
during tar or coping? If is bigger then "%id" there is possible mistake
in the disk partitioning.
I had these problem with 4096-Byte Sector Hard Drives, and it sound
very similar. ;-) Witch OS create the partitions? If windows, that can
be problem. ;-)


--
Honzik
------------------------------------------------------------------------
friedl_jan's Profile: http://forums.novell.com/member.php?userid=56174
View this thread: http://forums.novell.com/showthread.php?t=444967

albumns
21-Sep-2011, 20:26
friedl_jan;2139335 Wrote:
> Hi,
> i red all replies and have some idea ;-)
>
> Are you using tar operation on removable or on physical disk? With file
> system are you using?
>
> Do you anytime check the value "%wa = waiting operation" in the top
> during tar or coping? If is bigger then "%id" there is possible mistake
> in the disk partitioning.
> I had these problem with 4096-Byte Sector Hard Drives, and it sound
> very similar. ;-) Witch OS create the partitions? If windows, that can
> be problem. ;-)

Thank you very much for kind reply. I do the tar operation in physical
disk and I use ext3 which is the same with removable disk. someone
suggest me to use this command to check something, but I don't know
what's that:

sudo /sbin/hdparm -i /dev/sda
/dev/sda:

Model=ST9500420AS, FwRev=D005SDM1, SerialNo=5VJ5J20R
Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4
BuffType=unknown, BuffSize=16384kB, MaxMultSect=16, MultSect=16
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=976773168
IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6
AdvancedPM=yes: unknown setting WriteCache=enabled
Drive conforms to: unknown: ATA/ATAPI-4,5,6,7

* signifies the current active mode


--
albumns
------------------------------------------------------------------------
albumns's Profile: http://forums.novell.com/member.php?userid=48883
View this thread: http://forums.novell.com/showthread.php?t=444967

albumns
21-Sep-2011, 20:26
here is the output for extracting files:

/dev/sda:
Timing buffered disk reads: 124 MB in 3.07 seconds = 40.43 MB/sec


--
albumns
------------------------------------------------------------------------
albumns's Profile: http://forums.novell.com/member.php?userid=48883
View this thread: http://forums.novell.com/showthread.php?t=444967

albumns
21-Sep-2011, 20:26
here is the out put for command ''for i in {1..5};do hdparm -t
/dev/sda;done" when extracting something

/dev/sda:
Timing buffered disk reads: 130 MB in 3.02 seconds = 43.01 MB/sec

/dev/sda:
Timing buffered disk reads: 98 MB in 3.04 seconds = 32.28 MB/sec

/dev/sda:
Timing buffered disk reads: 64 MB in 3.03 seconds = 21.09 MB/sec

/dev/sda:
Timing buffered disk reads: 106 MB in 3.05 seconds = 34.71 MB/sec

/dev/sda:
Timing buffered disk reads: 62 MB in 3.00 seconds = 20.67 MB/sec


--
albumns
------------------------------------------------------------------------
albumns's Profile: http://forums.novell.com/member.php?userid=48883
View this thread: http://forums.novell.com/showthread.php?t=444967